Austin Rochford (Posts about NBA)

An Improved Analysis of NBA Foul Calls with Python

Austin Rochford — Sun, 04 Feb 2018 05:00:00 GMT

Last April, I wrote a post that used Bayesian item-response theory models to analyze NBA foul call data. Last November, I spoke about a greatly improved version of these models at PyData NYC. This post is a write-up of the models from that talk.

Last Two-minute Report

Since late in the 2014-2015 season, the NBA has issued last two minute reports. These reports give the league’s assessment of the correctness of foul calls and non-calls in the last two minutes of any game where the score difference was three or fewer points at any point in the last two minutes.

These reports are notably different from play-by-play logs, in that they include information on non-calls for notable on-court interactions. This non-call information presents a unique opportunity to study the factors that impact foul calls. There is a level of subjectivity inherent in the the NBA’s definition of notable on-court interactions which we attempt to mitigate later using season-specific factors.

Loading the data

Russel Goldenberg of The Pudding has been scraping the PDFs that the NBA publishes and transforming them into a CSV for some time. I am grateful for his work, which has enabled this analysis.

We download the data locally to be kind to GitHub.

%matplotlib inline

import datetime
from itertools import product
import logging
import pickle

from matplotlib import pyplot as plt
from matplotlib.offsetbox import AnchoredText
from matplotlib.ticker import FuncFormatter, StrMethodFormatter
import numpy as np
import pandas as pd
import scipy as sp
import seaborn as sns
from sklearn.preprocessing import LabelEncoder
from theano import tensor as tt

pct_formatter = StrMethodFormatter('{x:.1%}')

sns.set()
blue, green, *_ = sns.color_palette()

plt.rc('figure', figsize=(8, 6))

LABELSIZE = 14
plt.rc('axes', labelsize=LABELSIZE)
plt.rc('axes', titlesize=LABELSIZE)
plt.rc('figure', titlesize=LABELSIZE)
plt.rc('legend', fontsize=LABELSIZE)
plt.rc('xtick', labelsize=LABELSIZE)
plt.rc('ytick', labelsize=LABELSIZE)

SEED = 207183 # from random.org, for reproducibility

# keep theano from complaining about compile locks for small models
(logging.getLogger('theano.gof.compilelock')
        .setLevel(logging.CRITICAL))

%%bash
DATA_URI=https://raw.githubusercontent.com/polygraph-cool/last-two-minute-report/32f1c43dfa06c2e7652cc51ea65758007f2a1a01/output/all_games.csv
DATA_DEST=/tmp/all_games.csv

if [[ ! -e $DATA_DEST ]];
then
    wget -q -O $DATA_DEST $DATA_URI
fi

We use only a subset of the columns in the source data set.

USECOLS = [
    'period',
    'seconds_left',
    'call_type',
    'committing_player',
    'disadvantaged_player',
    'review_decision',
    'play_id',
    'away',
    'home',
    'date',
    'score_away',
    'score_home',
    'disadvantaged_team',
    'committing_team'
]

orig_df = pd.read_csv(
    '/tmp/all_games.csv',
    usecols=USECOLS,
    index_col='play_id',
    parse_dates=['date']
)

The data set contains more than 16,000 plays.

orig_df.shape[0]

Each row of the DataFrame represents a play and each column describes an attrbiute of the play:

period is the period of the game,
seconds_left is the number of seconds remaining in the game,
call_type is the type of call,
committing_player and disadvantaged_player are the names of the players involved in the play,
review_decision is the opinion of the league reviewer on whether or not the play was called correctly:
- review_decision = "INC" means the call was an incorrect noncall,
- review_decision = "CNC" means the call was an correct noncall,
- review_decision = "IC" means the call was an incorrect call, and
- review_decision = "CC" means the call was an correct call,
away and home are the abbreviations of the teams involved in the game,
date is the date on which the game was played,
score_away and score_home are the scores of the away and home team during the play, respectively, and
disadvantaged_team and committing_team indicate how each team is involved in the play.

orig_df.head(n=2).T

play_id	20150301CLEHOU-0	20150301CLEHOU-1
period	Q4	Q4
seconds_left	112	103
call_type	Foul: Shooting	Foul: Shooting
committing_player	Josh Smith	J.R. Smith
disadvantaged_player	Kevin Love	James Harden
review_decision	CNC	CC
away	CLE	CLE
home	HOU	HOU
date	2015-03-01 00:00:00	2015-03-01 00:00:00
score_away	103	103
score_home	105	105
disadvantaged_team	CLE	HOU
committing_team	HOU	CLE

Research questions

In this post, we answer two questions:

How does game context impact foul calls?
Is (not) committing and/or drawing fouls a measurable player skill?

The previous post focused on the second question, and gave the first question only a cursory treatment. This post enhances our treatment of the first question, in order to control for non-skill factors influencing foul calls (namely intentional fouls). Controlling for these factors makes our estimates of player skill more realistic.

Exploratory Data Analysis

First we examine the types of calls present in the data set.

(orig_df['call_type']
        .value_counts()
        .head(n=15))

Foul: Personal                     4736
Foul: Shooting                     4201
Foul: Offensive                    2846
Foul: Loose Ball                   1316
Turnover: Traveling                 779
Instant Replay: Support Ruling      607
Foul: Defense 3 Second              277
Instant Replay: Overturn Ruling     191
Foul: Personal Take                 172
Turnover: 3 Second Violation        139
Turnover: 24 Second Violation       126
Turnover: 5 Second Inbound           99
Stoppage: Out-of-Bounds              96
Violation: Lane                      84
Foul: Away from Play                 82
Name: call_type, dtype: int64

The portion of call_type before the colon is the general category of the call. We count the occurence of these categories below.

(orig_df['call_type']
        .str.split(':', expand=True)
        .iloc[:, 0]
        .value_counts()
        .plot(
            kind='bar',
            color=blue, logy=True, 
            title="Call types"
        )
        .set_ylabel("Frequency"));

We restrict our attention to foul calls, though other call types would be interesting to study in the future.

foul_df = orig_df[
    orig_df['call_type']
           .fillna("UNKNOWN")
           .str.startswith("Foul")
]

We count the foul call types below.

(foul_df['call_type']
        .str.split(': ', expand=True)
        .iloc[:, 1]
        .value_counts()
        .plot(
            kind='bar',
            color=blue, logy=True,
            title="Foul Types"
        )
        .set_ylabel("Frequency"));

We restrict our attention to the five foul types below, which generally involve two players. This subset of fouls allows us to pursue our second research question in the most direct manner.

FOULS = [
    f"Foul: {foul_type}"
        for foul_type in [
            "Personal",
            "Shooting",
            "Offensive",
            "Loose Ball",
            "Away from Play"
        ]
]

Data transformation

There are a number of misspelled team names in the data, which we correct.

TEAM_MAP = {
    "NKY": "NYK",
    "COS": "BOS",
    "SAT": "SAS",
    "CHi": "CHI",
    "LA)": "LAC",
    "AT)": "ATL",
    "ARL": "ATL"
}

def correct_team_name(col):
    def _correct_team_name(df):
        return df[col].apply(lambda team_name: TEAM_MAP.get(team_name, team_name))
    
    return _correct_team_name

We also convert each game date to an NBA season.

def date_to_season(date):
    if date >= datetime.datetime(2017, 10, 17):
        return '2017-2018'
    elif date >= datetime.datetime(2016, 10, 25):
        return '2016-2017'
    elif date >= datetime.datetime(2015, 10, 27):
        return '2015-2016'
    else:
        return '2014-2015'

We clean the data by

restricting to plays that occured during the last two minutes of regulation,
imputing incorrect noncalls when review_decision is missing,
correcting team names,
converting game dates to seasons,
restricting to the foul types discussed above,
restricting to the plays that happened during the 2015-2016 and 2016-2017 regular seasons (those are the only full seasons in the data set as of February 2018), and
dropping unneeded rows and columns.

clean_df = (foul_df.where(lambda df: df['period'] == "Q4")
                   .where(lambda df: (df['date'].between(datetime.datetime(2016, 10, 25),
                                                         datetime.datetime(2017, 4, 12))
                                     | df['date'].between(datetime.datetime(2015, 10, 27),
                                                          datetime.datetime(2016, 5, 30)))
                   )
                   .assign(
                       review_decision=lambda df: df['review_decision'].fillna("INC"),
                       committing_team=correct_team_name('committing_team'),
                       disadvantged_team=correct_team_name('disadvantaged_team'),
                       away=correct_team_name('away'),
                       home=correct_team_name('home'),
                       season=lambda df: df['date'].apply(date_to_season)
                   )
                   .where(lambda df: df['call_type'].isin(FOULS))
                   .dropna()
                   .drop('period', axis=1)
                   .assign(call_type=lambda df: (df['call_type']
                                                   .str.split(': ', expand=True)  
                                                   .iloc[:, 1])))

About 55% of the rows in the original data set remain.

clean_df.shape[0] / orig_df.shape[0]

0.5516564417177914

clean_df.head(n=2).T

play_id	20151028INDTOR-1	20151028INDTOR-2
seconds_left	89	73
call_type	Shooting	Shooting
committing_player	Ian Mahinmi	Bismack Biyombo
disadvantaged_player	DeMar DeRozan	Paul George
review_decision	CC	IC
away	IND	IND
home	TOR	TOR
date	2015-10-28 00:00:00	2015-10-28 00:00:00
score_away	99	99
score_home	106	106
disadvantaged_team	TOR	IND
committing_team	IND	TOR
disadvantged_team	TOR	IND
season	2015-2016	2015-2016

We use scikit-learn’s LabelEncoder to transform categorical features (call type, player, and season) to integers.

call_type_enc = LabelEncoder().fit(
    clean_df['call_type']
)
n_call_type = call_type_enc.classes_.size

player_enc = LabelEncoder().fit(
    np.concatenate((
        clean_df['committing_player'],
        clean_df['disadvantaged_player']
    ))
)
n_player = player_enc.classes_.size

season_enc = LabelEncoder().fit(
    clean_df['season']
)
n_season = season_enc.classes_.size

We transform the data by

rounding seconds_left to the nearest second (purely for convenience),
transforming categorical features to integer ids,
setting foul_called equal to one or zero depending on whether or not a foul was called, and
setting score_committing and score_disadvantaged to the score of the committing and disadvantaged teams, respectively.

df = (clean_df[['seconds_left']]
              .round(0)
              .assign(
                call_type=call_type_enc.transform(clean_df['call_type']),
                foul_called=1. * clean_df['review_decision'].isin(['CC', 'INC']),
                player_committing=player_enc.transform(clean_df['committing_player']),
                player_disadvantaged=player_enc.transform(clean_df['disadvantaged_player']),
                score_committing=clean_df['score_home'].where(
                    clean_df['committing_team'] == clean_df['home'],
                    clean_df['score_away']
                ),
                score_disadvantaged=clean_df['score_home'].where(
                    clean_df['disadvantaged_team'] == clean_df['home'],
                    clean_df['score_away']
                ),
                season=season_enc.transform(clean_df['season'])
              ))

The resulting data is ready for analysis.

df.head(n=2).T

play_id	20151028INDTOR-1	20151028INDTOR-2
seconds_left	89.0	73.0
call_type	4.0	4.0
foul_called	1.0	0.0
player_committing	162.0	36.0
player_disadvantaged	98.0	358.0
score_committing	99.0	106.0
score_disadvantaged	106.0	99.0
season	0.0	0.0

Modeling

We follow George Box’s modeling workflow, as summarized by Dustin Tran:

build a model of the science,
infer the model given data, and
criticize the model given data.

Baseline model

Build a model of the science

Below we examine the foul call rate by season.

def make_foul_rate_yaxis(ax, label="Observed foul call rate"):
    ax.yaxis.set_major_formatter(pct_formatter)
    ax.set_ylabel(label)
    
    return ax

make_foul_rate_yaxis(
    df.pivot_table('foul_called', 'season')
      .rename(index=season_enc.inverse_transform)
      .rename_axis("Season")
      .plot(kind='bar', rot=0, legend=False)
);

There is a pronounced difference between the foul call rate in the 2015-2016 and 2016-2017 NBA seasons; our first model accounts for this difference.

We use pymc3 to specify our models. Our first model is given by

\[ \begin{align*} \beta^{\textrm{season}}_s & \sim N(0, 5) \\ \eta^{\textrm{game}}_k & = \beta^{\textrm{season}}_{s(k)} \\ p_k & = \textrm{sigm}\left(\eta^{\textrm{game}}_k\right). \end{align*} \]

We use a logistic regression model with different factors for each season.

import pymc3 as pm

with pm.Model() as base_model:
    β_season = pm.Normal('β_season', 0., 5., shape=n_season)
    p = pm.Deterministic('p', pm.math.sigmoid(β_season))

Foul calls are Bernoulli trials, $y_k \sim \textrm{Bernoulli}(p_k).$

season = df['season'].values

with base_model:
    y = pm.Bernoulli(
        'y', p[season],
        observed=df['foul_called']
    )

Infer the model given data

We now sample from the model’s posterior distribution.

NJOBS = 3

SAMPLE_KWARGS = {
    'draws': 1000,
    'njobs': NJOBS,
    'random_seed': [
        SEED + i for i in range(NJOBS)
    ],
    'nuts_kwargs': {
        'target_accept': 0.9
    }
}

with base_model:
    base_trace = pm.sample(**SAMPLE_KWARGS)

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (3 chains in 3 jobs)
NUTS: [β_season]
100%|██████████| 1500/1500 [00:07<00:00, 198.65it/s]

Convergence diagnostics

We rely on three diagnostics to ensure that our samples have converged to the posterior distribution:

Energy plots: if the two distributions in the energy plot differ significantly (espescially in the tails), the sampling was not very efficient.
Bayesian fraction of missing information (BFMI): BFMI quantifies this difference with a number between zero and one. A BFMI close to (or exceeding) one is preferable, and a BFMI lower than 0.2 is indicative of efficiency issues.
Gelman-Rubin statistics: Gelman-Rubin statistics near one are preferable, and values less than 1.1 are generally taken to indicate convergence.

For more information on energy plots and BFMI consult Robust Statistical Workflow with PyStan.

bfmi = pm.bfmi(base_trace)

max_gr = max(
    np.max(gr_stats) for gr_stats in pm.gelman_rubin(base_trace).values()
)

CONVERGENCE_TITLE = lambda: f"BFMI = {bfmi:.2f}\nGelman-Rubin = {max_gr:.3f}"

(pm.energyplot(base_trace, legend=False, figsize=(6, 4))
   .set_title(CONVERGENCE_TITLE()));

Criticize the model given data

We use the samples from p’s posterior distribution to calculate residuals, which we use to criticize our models. These residuals allow us to assess how well our model describes the data-generation process and to discover unmodeled sources of variation.

base_trace['p']

array([[ 0.4052151 ,  0.30696232],
       [ 0.3937377 ,  0.30995026],
       [ 0.39881138,  0.29866616],
       ..., 
       [ 0.40279887,  0.31166828],
       [ 0.4077945 ,  0.30299785],
       [ 0.40207901,  0.29991789]])

resid_df = (df.assign(p_hat=base_trace['p'][:, df['season']].mean(axis=0))
              .assign(resid=lambda df: df['foul_called'] - df['p_hat']))

resid_df[['foul_called', 'p_hat', 'resid']].head()

	foul_called	p_hat	resid
play_id
20151028INDTOR-1	1.0	0.403875	0.596125
20151028INDTOR-2	0.0	0.403875	-0.403875
20151028INDTOR-3	1.0	0.403875	0.596125
20151028INDTOR-4	0.0	0.403875	-0.403875
20151028INDTOR-6	0.0	0.403875	-0.403875

The per-season residuals are quite small, which is to be expected.

(resid_df.pivot_table('resid', 'season')
         .rename(index=season_enc.inverse_transform))

	resid
season
2015-2016	-0.000162
2016-2017	-0.000219

Anyone who has watched a close basketball game will realize that we have neglected an important factor in late game foul calls — intentional fouls. Near the end of the game, intentional fouls are used by the losing team when they are on defense to end the leading team’s possession as quickly as possible.

The influence of intentional fouls in the plot below is shown by the rapidly increasing of the residuals as the number of seconds left in the game decreases.

def make_time_axes(ax,
                   xlabel="Seconds remaining in game",
                   ylabel="Observed foul call rate"):
    ax.invert_xaxis()
    ax.set_xlabel(xlabel)
    
    return make_foul_rate_yaxis(ax, label=ylabel)

make_time_axes(
    resid_df.pivot_table('resid', 'seconds_left')
            .reset_index()
            .plot('seconds_left', 'resid', kind='scatter'),
    ylabel="Residual"
);

Possession model

Build a model of the science

The following plot illustrates the fact that only the trailing team has any incentive to committ intentional fouls.

df['trailing_committing'] = (df['score_committing']
                               .lt(df['score_disadvantaged'])
                               .mul(1.)
                               .astype(np.int64))

make_time_axes(
    df.pivot_table(
        'foul_called',
        'seconds_left',
        'trailing_committing'
      )
      .rolling(20)
      .mean()
      .rename(columns={
          0: "No", 1: "Yes"
      })
      .rename_axis(
          "Committing team is trailing",
          axis=1
      )
      .plot()
);

Intentional fouls are only useful when the trailing (and committing) team is on defense. The plot below reflects this fact; shooting and personal fouls are almost always called against the defensive player; we see that they are called at a much higher rate than offensive fouls.

ax = (df.pivot_table('foul_called', 'call_type')
        .rename(index=call_type_enc.inverse_transform)
        .rename_axis("Call type", axis=0)
        .plot(kind='barh', legend=False))

ax.xaxis.set_major_formatter(pct_formatter);
ax.set_xlabel("Observed foul call rate");

We continue to model the differnce in foul call rates between seasons.

with pm.Model() as poss_model:
    β_season = pm.Normal('β_season', 0., 5., shape=2)

Throughout this post, we will use hierarchical distributions to model the variation of foul call rates. For much more information on hierarchical models, consult Data Analysis Using Regression and Multilevel/Hierarchical Models. We use the priors

\[ \begin{align*} \sigma_{\textrm{call}} & \sim \operatorname{HalfNormal}(5) \\ \beta^{\textrm{call}}_{c} & \sim \operatorname{Hierarchical-Normal}(0, \sigma_{\textrm{call}}^2). \end{align*} \]

For sampling efficiency, we use an [non-centered parametrization](http://twiecki.github.io/blog/2017/02/08/bayesian-hierchical-non-centered/#The-Funnel-of-Hell-(and-how-to-escape-it%29) of the hierarchical normal distribution.

def hierarchical_normal(name, shape, σ_shape=1):
    Δ = pm.Normal(f'Δ_{name}', 0., 1., shape=shape)
    σ = pm.HalfNormal(f'σ_{name}', 5., shape=σ_shape)
    
    return pm.Deterministic(name, Δ * σ)

Each call type has a different foul call rate.

with poss_model:
    β_call = hierarchical_normal('β_call', n_call_type)

We add score difference and the number of possessions by which the committing team is trailing to the DataFrame.

df['score_diff'] = (df['score_disadvantaged']
                      .sub(df['score_committing']))

df['trailing_poss'] = (df['score_diff']
                         .div(3)
                         .apply(np.ceil))

trailing_poss_enc = LabelEncoder().fit(df['trailing_poss'])
trailing_poss = trailing_poss_enc.transform(df['trailing_poss'])
n_trailing_poss = trailing_poss_enc.classes_.size

The plot below shows that the foul call rate (over time) varies based on the score difference (quantized into possessions) between the disadvanted team and the committing team. We assume that at most three points can be scored in a single possession (while this is not quite correct, four-point plays are rare enough that we do not account for them in our analysis).

make_time_axes(
    df.pivot_table(
        'foul_called',
        'seconds_left',
        'trailing_poss'
      )
      .loc[:, 1:3]
      .rolling(20).mean()
      .rename_axis(
          "Trailing possessions\n(committing team)",
          axis=1
      )
      .plot()
);

The plot below reflects the fact that intentional fouls are disproportionately personal fouls; the rate at which personal fouls are called increases drastically as the game nears its end.

make_time_axes(
    df.pivot_table('foul_called', 'seconds_left', 'call_type')
      .rolling(20).mean()
      .rename(columns=call_type_enc.inverse_transform)
      .rename_axis(None, axis=1)
      .plot()
);

Due to the NBA’s shot clock, the natural timescale of a basketball game is possessions, not seconds, remaining.

df['remaining_poss'] = (df['seconds_left']
                          .floordiv(25)
                          .add(1))

remaining_poss_enc = LabelEncoder().fit(df['remaining_poss'])
remaining_poss = remaining_poss_enc.transform(df['remaining_poss'])
n_remaining_poss = remaining_poss_enc.classes_.size

Below we plot the foul call rate across trailing possession/remaining posession pairs. Note that we always calculate trailing possessions (trailing_poss) from the perspective of the committing team. For instance, trailing_poss = 1 indicates that the committing team is trailing by 1-3 points, whereas trailing_poss = -1 indicates that the committing team is leading by 1-3 points.

ax = sns.heatmap(
    df.pivot_table(
        'foul_called',
        'trailing_poss',
        'remaining_poss'
      )
      .rename_axis(
          "Trailing possessions\n(committing team)",
          axis=0
      )
      .rename_axis("Remaining possessions", axis=1),
    cmap='seismic',
    cbar_kws={'format': pct_formatter}
)

ax.invert_yaxis();
ax.set_title("Observed foul call rate");

The heatmap above shows that the foul call rate increases significantly when the committing team is trailing by more than the number of possessions remaining in the game. That is, teams resort to intentional fouls only when the opposing team can run out the clock and guarantee a win. (Since we have quantized the score difference and time into posessions, this conclusion is not entirely correct; it is, however, correct enough for our purposes.)

call_name_df = df.assign(
    call_type=lambda df: call_type_enc.inverse_transform(
        df['call_type'].values
    )
)

diff_df = (pd.merge(
                call_name_df,
                call_name_df.groupby('call_type')
                            ['foul_called']
                            .mean()
                            .rename('avg_foul_called')
                            .reset_index()
             )
             .assign(diff=lambda df: df['foul_called'] - df['avg_foul_called']))

The heatmaps below are broken out by call type, and show the difference between the foul call rate for each trailing/remaining possession combination and the overall foul call rate for the call type in question

def plot_foul_diff_heatmap(*_, data=None, **kwargs):
    ax = plt.gca()

    sns.heatmap(
        data.pivot_table(
            'diff',
            'trailing_poss',
            'remaining_poss'
        ),
        cmap='seismic', robust=True,
        cbar_kws={'format': pct_formatter}
    )
    
    ax.invert_yaxis()
    ax.set_title("Observed foul call rate")

(sns.FacetGrid(diff_df, col='call_type', col_wrap=3, aspect=1.5)
    .map_dataframe(plot_foul_diff_heatmap)
    .set_axis_labels(
        "Remaining possessions",
        "Trailing possessions\n(committing team)"
    )
    .set_titles("{col_name}"));

These plots confirm that most intentional fouls are personal fouls. They also show that the three-way interaction between trailing possesions, remaining possessions, and call type are important to model foul call rates.

\[ \begin{align*} \sigma_{\textrm{poss}, c} & \sim \operatorname{HalfNormal}(5) \\ \beta^{\textrm{poss}}_{t, r, c} & \sim \operatorname{Hierarchical-Normal}(0, \sigma_{\textrm{poss}, c}^2) \end{align*} \]

with poss_model:
    β_poss = hierarchical_normal(
        'β_poss',
        (n_trailing_poss, n_remaining_poss, n_call_type),
        σ_shape=(1, 1, n_call_type)
    )

The foul call rate is a combination of season, call type, and possession factors.

\[\eta^{\textrm{game}}_k = \beta^{\textrm{season}}_{s(k)} + \beta^{\textrm{call}}_{c(k)} + \beta^{\textrm{poss}}_{t(k),r(k),c(k)}\]

call_type = df['call_type'].values

with poss_model:
    η_game = β_season[season] \
                + β_call[call_type] \
                + β_poss[
                    trailing_poss,
                    remaining_poss,
                    call_type
                ]

\[ \begin{align*} p_k & = \operatorname{sigm}\left(\eta^{\textrm{game}}_k\right) \end{align*} \]

with poss_model:
    p = pm.Deterministic('p', pm.math.sigmoid(η_game))
    y = pm.Bernoulli('y', p, observed=df['foul_called'])

Infer the model given data

Again, we sample from the model’s posterior distribution.

with poss_model:
    poss_trace = pm.sample(**SAMPLE_KWARGS)

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (3 chains in 3 jobs)
NUTS: [σ_β_poss_log__, Δ_β_poss, σ_β_call_log__, Δ_β_call, β_season]
100%|██████████| 1500/1500 [07:56<00:00,  3.15it/s]
There were 5 divergences after tuning. Increase `target_accept` or reparameterize.
There were 10 divergences after tuning. Increase `target_accept` or reparameterize.
There were 9 divergences after tuning. Increase `target_accept` or reparameterize.
The number of effective samples is smaller than 25% for some parameters.

The BFMI and Gelman-Rubin statistics for this model indicate no problems with sampling and good convergence.

bfmi = pm.bfmi(poss_trace)

max_gr = max(
    np.max(gr_stats) for gr_stats in pm.gelman_rubin(poss_trace).values()
)

(pm.energyplot(poss_trace, legend=False, figsize=(6, 4))
   .set_title(CONVERGENCE_TITLE()));

Criticize the model given data

Again, we calculate residuals.

resid_df = (df.assign(p_hat=poss_trace['p'].mean(axis=0))
              .assign(resid=lambda df: df.foul_called - df.p_hat))

The following plots show that, grouped various ways, the residuals for this model are relatively well-distributed.

ax = sns.heatmap(
    resid_df.pivot_table(
                'resid',
                'trailing_poss',
                'remaining_poss'
            )
            .rename_axis(
                "Trailing possessions\n(committing team)",
                axis=0
            )
            .rename_axis(
                "Remaining possessions",
                axis=1
            )
            .loc[-3:3],
    cmap='seismic', 
    cbar_kws={'format': pct_formatter}
)

ax.invert_yaxis();
ax.set_title("Observed foul call rate");

N_BIN = 20

bin_ix, bins = pd.qcut(
    resid_df.p_hat, N_BIN,
    labels=np.arange(N_BIN),
    retbins=True
)

ax = (resid_df.groupby(bins[bin_ix])
              .resid.mean()
              .rename_axis('p_hat', axis=0)
              .reset_index()
              .plot('p_hat', 'resid', kind='scatter'))

ax.xaxis.set_major_formatter(pct_formatter);
ax.set_xlabel(r"Binned $\hat{p}$");

make_foul_rate_yaxis(ax, label="Residual");

ax = (resid_df.groupby('seconds_left')
              .resid.mean()
              .reset_index()
              .plot('seconds_left', 'resid', kind='scatter'))
make_time_axes(ax, ylabel="Residual");

Model selection

Now that we have two models, we can engage in model selection. We use the widely applicable Bayesian information criterion (WAIC) for model selection.

MODEL_NAME_MAP = {
    0: "Base",
    1: "Possession"
}

comp_df = (pm.compare(
                (base_trace, poss_trace),
                (base_model, poss_model)
             )
             .rename(index=MODEL_NAME_MAP)
             .loc[MODEL_NAME_MAP.values()])

Since smaller WAICs are better, the possession model clearly outperforms the base model.

comp_df

	WAIC	pWAIC	dWAIC	weight	SE	dSE	var_warn
Base	11610.1	2.11	1541.98	0	56.9	73.43	0
Possession	10068.1	82.93	0	1	88.05	0	0

fig, ax = plt.subplots()

ax.errorbar(
    np.arange(len(MODEL_NAME_MAP)),
    comp_df.WAIC,
    yerr=comp_df.SE, fmt='o'
);

ax.set_xticks(np.arange(len(MODEL_NAME_MAP)));
ax.set_xticklabels(comp_df.index);
ax.set_xlabel("Model");

ax.set_ylabel("WAIC");

Player item-response theory model

Build a model of the science

We now turn to the question of whether or not committing and/or drawing fouls is a measurable skill. We use an item-response theory (IRT) model to study this question. For more information on Bayesian item-response models, consult the following references.

Practical Issues in Implementing and Understanding Bayesian Ideal Point Estimation is an excellent introduction to applied Bayesian IRT models and has inspired much of this work.
Bayesian Item Response Modeling — Theory and Applications is a comprehensive mathematical overview of Bayesien IRT modeling.

The item-response theory model includes the season, call type, and possession terms of the previous models.

with pm.Model() as irt_model:
    β_season = pm.Normal('β_season', 0., 5., shape=n_season)
    β_call = hierarchical_normal('β_call', n_call_type)
    β_poss = hierarchical_normal(
        'β_poss',
        (n_trailing_poss, n_remaining_poss, n_call_type),
        σ_shape=(1, 1, n_call_type)
    )
    
    η_game = β_season[season] \
                + β_call[call_type] \
                + β_poss[
                    trailing_poss,
                    remaining_poss,
                    call_type
                ]

Each disadvantaged player has an ideal point (per season).

\[ \begin{align*} \sigma_{\theta} & \sim \operatorname{HalfNormal}(5) \\ \theta^{\textrm{player}}_{i, s} & \sim \operatorname{Hierarchical-Normal}(0, \sigma_{\theta}^2) \end{align*} \]

player_disadvantaged = df['player_disadvantaged'].values
n_player = player_enc.classes_.size

with irt_model:
    θ_player = hierarchical_normal(
        'θ_player', (n_player, n_season)
    )
    θ = θ_player[player_disadvantaged, season]

Each committing player has an ideal point (per season).

\[ \begin{align*} \sigma_{b} & \sim \operatorname{HalfNormal}(5) \\ b^{\textrm{player}}_{j, s} & \sim \operatorname{Hierarchical-Normal}(0, \sigma_{b}^2) \end{align*} \]

player_committing = df['player_committing'].values

with irt_model:
    b_player = hierarchical_normal(
        'b_player', (n_player, n_season)
    )
    b = b_player[player_committing, season]

Players affect the foul call rate through the difference in their ideal points.

\[\eta^{\textrm{player}}_k = \theta_k - b_k\]

with irt_model:
    η_player = θ - b

The sum of the game and player effects determines the foul call probability.

\[\eta_k = \eta^{\textrm{game}}_k + \eta^{\textrm{player}}_k\]

with irt_model:
    η = η_game + η_player

with irt_model:
    p = pm.Deterministic('p', pm.math.sigmoid(η))
    y = pm.Bernoulli(
        'y', p,
        observed=df['foul_called']
    )

Infer the model given data

Again, we sample from the model’s posterior distribution.

with irt_model:
    irt_trace = pm.sample(**SAMPLE_KWARGS)

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (3 chains in 3 jobs)
NUTS: [σ_b_player_log__, Δ_b_player, σ_θ_player_log__, Δ_θ_player, σ_β_poss_log__, Δ_β_poss, σ_β_call_log__, Δ_β_call, β_season]
100%|██████████| 1500/1500 [13:55<00:00,  1.80it/s]
There were 3 divergences after tuning. Increase `target_accept` or reparameterize.
There were 1 divergences after tuning. Increase `target_accept` or reparameterize.
There were 4 divergences after tuning. Increase `target_accept` or reparameterize.
The estimated number of effective samples is smaller than 200 for some parameters.

None of the sampling diagnostics indicate problems with convergence.

bfmi = pm.bfmi(irt_trace)

max_gr = max(
    np.max(gr_stats) for gr_stats in pm.gelman_rubin(irt_trace).values()
)

(pm.energyplot(irt_trace, legend=False, figsize=(6, 4))
   .set_title(CONVERGENCE_TITLE()));

Criticize the model given data, take three

The binned residuals for this model are more asymmetric than for the previous models, but still not too bad.

resid_df = (df.assign(p_hat=irt_trace['p'].mean(axis=0))
              .assign(resid=lambda df: df['foul_called'] - df['p_hat']))

N_BIN = 50

bin_ix, bins = pd.qcut(
    resid_df.p_hat, N_BIN,
    labels=np.arange(N_BIN),
    retbins=True
)

ax = (resid_df.groupby(bins[bin_ix])
              .resid.mean()
              .rename_axis('p_hat', axis=0)
              .reset_index()
              .plot('p_hat', 'resid', kind='scatter'))

ax.xaxis.set_major_formatter(pct_formatter);
ax.set_xlabel(r"Binned $\hat{p}$");

make_foul_rate_yaxis(ax, label="Residual");

ax = (resid_df.groupby('seconds_left')
              .resid.mean()
              .reset_index()
              .plot('seconds_left', 'resid', kind='scatter'))
make_time_axes(ax, ylabel="Residual");

Model selection

The IRT model is a marginal improvement over the possession model in terms of WAIC.

MODEL_NAME_MAP[2] = "IRT"

comp_df = (pm.compare(
                (base_trace, poss_trace, irt_trace),
                (base_model, poss_model, irt_model)
             )
             .rename(index=MODEL_NAME_MAP)
             .loc[MODEL_NAME_MAP.values()])

comp_df

	WAIC	pWAIC	dWAIC	weight	SE	dSE
Base	11610.1	2.11	1566.92	0	56.9	74.03
Possession	10068.1	82.93	24.94	0.08	88.05	10.99
IRT	10043.2	216.6	0	0.91	88.47	0

fig, ax = plt.subplots()
ax.errorbar(
    np.arange(len(MODEL_NAME_MAP)), comp_df.WAIC,
    yerr=comp_df.SE, fmt='o'
);
ax.set_xticks(np.arange(len(MODEL_NAME_MAP)));
ax.set_xticklabels(comp_df.index);
ax.set_xlabel("Model");
ax.set_ylabel("WAIC");

We now produce two DataFrames containing the estimated player ideal points per season.

def varname_to_param(varname):
    return varname[0]

def varname_to_player(varname):
    return int(varname[3:-2])

def varname_to_season(varname):
    return int(varname[-1])

irt_df = (pm.trace_to_dataframe(
                irt_trace, varnames=['θ_player', 'b_player']
            )
            .rename(columns=lambda col: col.replace('_player', ''))
            .T
            .apply(
               lambda s: pd.Series.describe(
                   s, percentiles=[0.055, 0.945]
               ),
               axis=1
            )
            [['mean', '5.5%', '94.5%']]
            .rename(columns={
                 '5.5%': 'low',
                 '94.5%': 'high'
            })
            .rename_axis('varname')
            .reset_index()
            .assign(
                param=lambda df: df['varname'].apply(varname_to_param),
                player=lambda df: df['varname'].apply(varname_to_player),
                season=lambda df: df['varname'].apply(varname_to_season)
            )
            .drop('varname', axis=1))

irt_df.head()

	mean	low	high	param	player	season
0	-0.016516	-0.323127	0.272022	θ	0	0
1	0.003845	-0.289842	0.290248	θ	0	1
2	0.008519	-0.289715	0.304574	θ	1	0
3	0.031367	-0.240853	0.339297	θ	1	1
4	-0.037530	-0.320010	0.226110	θ	2	0

player_irt_df = irt_df.pivot_table(
    index='player',
    columns=['param', 'season'],
    values='mean'
)

player_irt_df.head()

param	b		θ
season	0	1	0	1
player
0	-0.069235	0.013339	-0.016516	0.003845
1	-0.003003	0.001853	0.008519	0.031367
2	0.084515	0.089058	-0.037530	-0.002373
3	-0.028946	0.004360	0.003514	-0.000334
4	-0.001976	0.280380	0.072932	0.005571

The following plot shows that the committing skill appears to be somewhat larger than the disadvantaged skill. This difference seems reasonable because most fouls are committed by the player on defense; committing skill is quite likely to be correlated with defensive ability.

def plot_latent_params(df):
    fig, ax = plt.subplots()
    
    n, _ = df.shape
    y = np.arange(n)

    ax.errorbar(
        df['mean'], y,
        xerr=(df[['high', 'low']]
                .sub(df['mean'], axis=0)
                .abs()
                .values.T),
        fmt='o'
    )

    ax.set_yticks(y)
    ax.set_yticklabels(
        player_enc.inverse_transform(df.player)
    )
    ax.set_ylabel("Player")
    
    return fig, ax

fig, axes = plt.subplots(
    ncols=2, nrows=2, sharex=True,
    figsize=(16, 8)
)
(θ0_ax, θ1_ax), (b0_ax, b1_ax) = axes

bins = np.linspace(
    0.9 * irt_df['mean'].min(),
    1.1 * irt_df['mean'].max(),
    75
)

θ0_ax.hist(
    player_irt_df['θ', 0],
    bins=bins, normed=True
);
θ1_ax.hist(
    player_irt_df['θ', 1],
    bins=bins, normed=True
);

θ0_ax.set_yticks([]);
θ0_ax.set_title(
    r"$\hat{\theta}$ (" + season_enc.inverse_transform(0) + ")"
);

θ1_ax.set_yticks([]);
θ1_ax.set_title(
    r"$\hat{\theta}$ (" + season_enc.inverse_transform(1) + ")"
);

b0_ax.hist(
    player_irt_df['b', 0],
    bins=bins, normed=True, color=green
);
b1_ax.hist(
    player_irt_df['b', 1],
    bins=bins, normed=True, color=green
);

b0_ax.set_xlabel(
    r"$\hat{b}$ (" + season_enc.inverse_transform(0) + ")"
);

b0_ax.invert_yaxis();
b0_ax.xaxis.tick_top();
b0_ax.set_yticks([]);

b1_ax.set_xlabel(
    r"$\hat{b}$ (" + season_enc.inverse_transform(1) + ")"
);

b1_ax.invert_yaxis();
b1_ax.xaxis.tick_top();
b1_ax.set_yticks([]);

fig.suptitle("Disadvantaged skill", size=18);
fig.text(0.45, 0.02, "Committing skill", size=18)
fig.tight_layout();

We now examine the top and bottom ten players in each ability, across both seasons.

The top players in terms of disadvantaged ability tend to be good scorers (Jimmy Butler, Ricky Rubio, John Wall, Andre Iguodala). The presence of DeAndre Jordan in the top ten may to be due to the hack-a-Shaq phenomenon. In future work, it would be interesting to control for the disavantage player’s free throw percentage in order to mitigate the influence of the hack-a-Shaq effect on the measurement of latent skill.

Interestingly, the bottom players (in terms of disadvantaged ability) include many stars (Pau Gasol, Carmelo Anthony, Kevin Durant, Kawhi Leonard). The presence of these stars in the bottom may somewhat counteract the pervasive narrative that referees favor stars in their foul calls.

top_bot_irt_df = (irt_df.groupby('param')
                    .apply(
                        lambda df: pd.concat((
                                df.nlargest(10, 'mean'),
                                df.nsmallest(10, 'mean')
                            ),
                            axis=0, ignore_index=True
                        )
                    )
                    .reset_index(drop=True))

top_bot_irt_df.head()

	mean	low	high	param	player	season
0	0.351946	-0.026786	0.762273	b	86	0
1	0.320737	-0.027064	0.713128	b	23	0
2	0.280380	-0.071020	0.695970	b	4	1
3	0.279678	-0.057249	0.647667	b	462	1
4	0.271735	-0.106795	0.676231	b	78	0

fig, ax = plot_latent_params(
    top_bot_irt_df[top_bot_irt_df['param'] == 'θ']
                  .sort_values('mean')
)
ax.set_xlabel(r"$\hat{\theta}$");
ax.set_title("Top and bottom ten");

The top ten players in terms of committing skill include many defensive standouts (Danny Green — twice, Gordon Hayward, Paul George).

The bottom ten players include many that are known to be defensively challenged (Ricky Rubio and James Harden). Dwight Howard was, at one point, a fierce defender of the rim, but was well past his prime in 2015, when our data set begins. Chris Paul’s presence in the bottom is somewhat surprising.

fig, ax = plot_latent_params(
    top_bot_irt_df[top_bot_irt_df['param'] == 'b']
                  .sort_values('mean')
)
ax.set_xlabel(r"$\hat{b}$");
ax.set_title("Top and bottom ten");

In the sports analytics community, year-over-year correlation of latent parameters is the test of whether or not a latent quantity truly measures a skill. The following plots show a slight year-over-year correlation in the committing skill, but not much correlation in the disadvantaged skill.

def p_val_to_asterisks(p_val):
    if p_val < 0.0001:
        return "****"
    elif p_val < 0.001:
        return "***"
    elif p_val < 0.01:
        return "**"
    elif p_val < 0.05:
        return "*"
    else:
        return ""

def plot_corr(x, y, **kwargs):
    corrcoeff, p_val = sp.stats.pearsonr(x, y)
    asterisks = p_val_to_asterisks(p_val)
    
    artist = AnchoredText(
        f'{corrcoeff:.2f}{asterisks}',
        loc=10, frameon=False,
        prop=dict(size=LABELSIZE)
    )
    plt.gca().add_artist(artist)
    plt.grid(b=False)

PARAM_MAP = {
    'θ': r"$\hat{\theta}$",
    'b': r"$\hat{b}$"
}

def replace_label(label):
    param, season = eval(label)

    return "{param}\n({season})".format(
        param=PARAM_MAP[param],
        season=season_enc.inverse_transform(season)
    )
    
def style_grid(grid):
    for ax in grid.axes.flat:
        ax.grid(False)
        ax.set_xticklabels([]);
        ax.set_yticklabels([]);
        
        if ax.get_xlabel():
            ax.set_xlabel(replace_label(ax.get_xlabel()))

        if ax.get_ylabel():
            ax.set_ylabel(replace_label(ax.get_ylabel()))
            
    return grid

player_all_season = set(df.groupby('player_disadvantaged')
                          .filter(lambda df: df['season'].nunique() == n_season)
                          ['player_committing']) \
                        & set(df.groupby('player_committing')
                                .filter(lambda df: df['season'].nunique() == n_season)
                                ['player_committing'])

style_grid(
    sns.PairGrid(
        player_irt_df.loc[player_all_season],
        size=1.75
       )
       .map_upper(plt.scatter, alpha=0.5)
       .map_diag(plt.hist)
       .map_lower(plot_corr)
);

Since we can only reasonably estimate the skills of players for which we have sufficient foul call data, we plot the correlations below for players that appeared in at least ten plays in each season.

MIN = 10

player_has_min = set(df.groupby('player_disadvantaged')
                       .filter(lambda df: (df['season']
                                             .value_counts()
                                             .gt(MIN)
                                             .all()))
                       ['player_committing']) \
                    & set(df.groupby('player_committing')
                            .filter(lambda df: (df['season']
                                                  .value_counts()
                                                  .gt(MIN)
                                                  .all()))
                            ['player_committing'])

grid = style_grid(
    sns.PairGrid(
        player_irt_df.loc[player_has_min],
        size=1.75
       )
       .map_upper(plt.scatter, alpha=0.5)
       .map_diag(plt.hist)
       .map_lower(plot_corr)
)

As expected, the season-over-season latent skill correlations are higher for this subset of players.

From this figure, it seems that committing skill ($\hat{b}$) exists and is measurable, but is fairly small. It also seems that disadvantaged skill ($\hat{\theta}$), if it exists, is difficult to measure from this data set. Ideally, the NBA would release a foul report for the entirety of every game, but that seems quite unlikely.

In the future it would be useful to include a correction for the probability that a given game appears in the data set. This correction would help with the sample bias introduced by the fact that only games that are close in the last two minutes are included in the NBA’s reports.

This post is available as a Jupyter notebook here.

NBA Foul Calls and Bayesian Item Response Theory

Austin Rochford — Tue, 04 Apr 2017 04:00:00 GMT

An improved version of this analysis is available here.

(Author’s note: many thanks to Robert ([@atlhawksfanatic](https://twitter.com/atlhawksfanatic) on Twitter) for pointing out some subtleties in the data set that I had missed. This post has been revised in line with his feedback. Robert has a very interesting post about how last two minute refereeing has changed over the last three years; I highly recommend you read it.)

I recently found a very interesting data set derived from the NBA’s Last Two Minute Report by Russell Goldenberg of The Pudding. Since 2015, the NBA has released a report reviewing every call and non-call in the final two minutes of every NBA game where the teams were separated by five points or less with two minutes remaining. This data set has extracted each play from the NBA-distributed PDF and augmented it with information from Basketball Reference to produce a convenient CSV. The Pudding has published two very interesting visual essays using this data that you should definitely explore.

The NBA is certainly marketed as a star-centric league, so this data set presents a fantastic opportunity to understand the extent to which the players involved in a decision impact whether or not a foul is called. We will also explore other factors related to foul calls.

%matplotlib inline

import datetime
from warnings import filterwarnings

from matplotlib import pyplot as plt
from matplotlib.ticker import FuncFormatter
import numpy as np
import pandas as pd
import pymc3 as pm
from scipy.special import expit
import seaborn as sns

blue, green, red, purple, gold, teal = sns.color_palette()

million_dollars_formatter = FuncFormatter(lambda value, _: '${:.1f}M'.format(value / 1e6))
pct_formatter = FuncFormatter(lambda prop, _: "{:.1%}".format(prop))

filterwarnings('ignore', 'findfont')

Loading and preprocessing the data

We begin by loading the data set from GitHub. For reproducibility, we load the data from the most recent commit as of the time this post was published.

DATA_URI = 'https://raw.githubusercontent.com/polygraph-cool/last-two-minute-report/1b89b71df060add5538b70d391d7ad82a4c24db2/output/all_games.csv'

raw_df = (pd.read_csv(DATA_URI,
                      usecols=['committing_player', 'disadvantaged_player',
                               'committing_team', 'disadvantaged_team',
                               'seconds_left', 'review_decision', 'date'],
                      parse_dates=['date'])
            .where(lambda df: df.date >= datetime.datetime(2016, 10, 25))
            .dropna(subset=['date'])
            .drop('date', axis=1))
raw_df['review_decision'] = raw_df.review_decision.fillna("INC")
raw_df = (raw_df.dropna()
                .reset_index(drop=True))

We restrict our attention to decisions from the 2016-2017 NBA season, for which salary information is readily available from Basketball Reference.

raw_df.head()

	seconds_left	committing_player	disadvantaged_player	review_decision	disadvantaged_team	committing_team
0	102.0	Al-Farouq Aminu	George Hill	CNC	UTA	POR
1	98.0	Boris Diaw	Damian Lillard	CC	POR	UTA
2	64.0	Ed Davis	George Hill	CNC	UTA	POR
3	62.0	Rudy Gobert	CJ McCollum	INC	POR	UTA
4	27.1	CJ McCollum	Rodney Hood	CC	UTA	POR

We have only loaded some of the data set’s columns; see the original CSV header for the rest.

The response variable in our analysis is derived from review_decision, which contains information about whether the incident was a call or non-call and whether, upon post-game review, the NBA deemed the (non-)call correct or incorrect. Below we show the frequencies of each type of review_decision.

ax = (raw_df.groupby('review_decision')
            .size()
            .plot(kind='bar'))

ax.set_ylabel("Frequency");

The possible values of review_decision are

CC for correct call,
CNC for correct non-call,
IC for incorrect call, and
INC for incorrect non-call.

While review_decision decision provides information about both whether or not a foul was called and whether or not a foul was actually committed, this analysis will focus only on whether or not a foul was called. Including whether or not a foul was actually committed in this analysis introduces some subtleties that are best left to a future post.

In this dataset, the “committing” player is the one that a foul would be called against, if a foul was called on the play, and the other player is “disadvantaged.”

We now encode the data. Since the committing player on one play may be the disadvantaged player on another play, we melt the raw data frame to have one row per player-play combination so that we can encode the players in a way that is consistent across columns.

PLAYER_MAP = {
    "Jose Juan Barea": "JJ Barea",
    "Nene Hilario": "Nene",
    "Tim Hardaway": "Tim Hardaway Jr",
    "James Ennis": "James Ennis III",
    "Kelly Oubre": "Kelly Oubre Jr",
    "Taurean Waller-Prince": "Taurean Prince",
    "Glenn Robinson": "Glenn Robinson III",
    "Otto Porter": "Otto Porter Jr"
}

TEAM_MAP = {
    "NKY": "NYK",
    "COS": "BOS",
    "SAT": "SAS"
}

long_df = (pd.melt(
                (raw_df.reset_index(drop=True)
                       .rename_axis('play_id')
                       .reset_index()),
                id_vars=['play_id', 'review_decision',
                         'committing_team', 'disadvantaged_team',
                         'seconds_left'],
                value_vars=['committing_player', 'disadvantaged_player'],
                var_name='player', value_name='player_name_')
             # fix inconsistent player names
             .assign(player_name=lambda df: (df.player_name_
                                               .str.replace('\.', '')
                                               .apply(lambda name: PLAYER_MAP.get(name, name))))
             .assign(team_=lambda df: (df.committing_team
                                         .where(df.player == 'committing_player',
                                                df.disadvantaged_team)))
             # fix typos in team names
             .assign(team=lambda df: df.team_.apply(lambda team: TEAM_MAP.get(team, team)))
             .drop(['committing_team', 'disadvantaged_team', 'team_'], axis=1))

long_df['player_id'], player_map = long_df.player_name.factorize()

long_df.head()

	play_id	review_decision	seconds_left	player	player_name_	player_name	team	player_id
0	0	CNC	102.0	committing_player	Al-Farouq Aminu	Al-Farouq Aminu	POR	0
1	1	CC	98.0	committing_player	Boris Diaw	Boris Diaw	UTA	1
2	2	CNC	64.0	committing_player	Ed Davis	Ed Davis	POR	2
3	3	INC	62.0	committing_player	Rudy Gobert	Rudy Gobert	UTA	3
4	4	CC	27.1	committing_player	CJ McCollum	CJ McCollum	POR	4

After encoding, we pivot back to a wide data frame with one row per play.

df = (long_df.pivot_table(index=['play_id', 'review_decision', 'seconds_left'],
                          columns='player', values='player_id')
             .rename(columns={
                 'committing_player': 'committing_id',
                 'disadvantaged_player': 'disadvantaged_id'
             })
             .rename_axis('', axis=1)
             .reset_index()
             .assign(foul_called=lambda df: 1 * (df.review_decision.isin(['CC', 'IC'])))
             .drop(['play_id', 'review_decision'],
                   axis=1))

In addition to encoding the players, we have include a column (foul_called) that indicates whether or not a foul was called on the play.

df.head()

	seconds_left	committing_id	disadvantaged_id	foul_called
0	102.0	0	300	0
1	98.0	1	124	1
2	64.0	2	300	0
3	62.0	3	4	0
4	27.1	4	6	1

In order to understand how foul calls vary systematically across players, we will use salary as a proxy for “star power.” The salary data we use was downloaded from Basketball Reference.

SALARY_URI = 'http://www.austinrochford.com/resources/nba_irt/2016_2017_salaries.csv'

salary_df = (pd.read_csv(SALARY_URI, skiprows=1,
                         usecols=['Player', '2016-17'])
               .assign(player_name=lambda df: (df.Player
                                                 .str.split('\\', expand=True)[0]
                                                 .str.replace('\.', '')
                                                 # fix inconsistent player names
                                                 .apply(lambda name: PLAYER_MAP.get(name, name))),
                       salary=lambda df: (df['2016-17'].str
                                                       .lstrip('$')
                                                       .astype(np.float64)))
               .assign(log_salary=lambda df: np.log10(df.salary))
               .assign(std_log_salary=lambda df: (df.log_salary - df.log_salary.mean()) / df.log_salary.std())
               .drop(['Player', '2016-17'], axis=1)
               .groupby('player_name')
               .max()
               .select(lambda name: name in player_map)
               .assign(player_id=lambda df: (np.equal
                                               .outer(player_map, df.index)
                                               .argmax(axis=0)))
               .reset_index()
               .set_index('player_id')
               .sort_index())

Since NBA salaries span many orders of magnitude (LeBron James’ salary is just shy of $31M while the lowest paid player made just more than $200K) we will use log salaries, standardized to have mean zero and standard deviation one in our model.

salary_df.head()

	player_name	salary	log_salary	std_log_salary
player_id
0	Al-Farouq Aminu	7680965.0	6.885416	0.848869
1	Boris Diaw	7000000.0	6.845098	0.797879
2	Ed Davis	6666667.0	6.823909	0.771080
3	Rudy Gobert	2121288.0	6.326600	0.142129
4	CJ McCollum	3219579.0	6.507799	0.371293

We also produce a dataframe associating players to teams, along with some useful per-player summaries.

team_player_map = (long_df.groupby('team')
                          .player_id
                          .apply(pd.Series.drop_duplicates)
                          .reset_index(level=-1, drop=True)
                          .reset_index()
                          .assign(name=lambda df: player_map[df.player_id],
                                  disadvantaged_rate=lambda tpm_df: (df.groupby('disadvantaged_id')
                                                                       .foul_called
                                                                       .mean()
                                                                       .ix[tpm_df.player_id]
                                                                       .values),
                                  disadvantaged_plays=lambda tpm_df: (df.groupby('disadvantaged_id')
                                                                        .size()
                                                                        .ix[tpm_df.player_id]
                                                                        .values))
                          .fillna(0))

team_player_map.head()

	team	player_id	disadvantaged_plays	disadvantaged_rate	name
0	ATL	114	8.0	0.000000	Kyle Korver
1	ATL	115	13.0	0.538462	Dwight Howard
2	ATL	116	44.0	0.272727	Paul Millsap
3	ATL	117	60.0	0.283333	Dennis Schroder
4	ATL	181	25.0	0.200000	Kent Bazemore

Modeling

Throughout this post, we will develop a series of models for understanding how foul calls vary across players, starting with a simple beta-Bernoulli model and working our way up to a hierachical item-response theory regression model.

Before building models, we must introduce a bit of notation. The index $i$ will correspond to a disadvantaged player and the index $j$ corresponds to a committing player. The index $k$ corresponds to a play. With this notation $i(k)$ and $j(k)$ are the index of the disadvantaged and committing player involved in play $k$, respectively. The binary variable $y_k$ indicates whether or not a foul was called on play $k$. All of our models use the likelihood

\[y_k \sim \textrm{Bernoulli}(p_{i(k), j(k)}).\]

Each model differs in its specification of the probability that a foul is called, $p_{i, j}$.

Beta-Bernoulli model

One of the simplest possible models for this data focuses only on the disadvantaged player, so $p_{i, j} = p_i$, and places independent beta priors on each $p_i$. For simplicity, we begin with uniform priors, $p_i \sim \textrm{Beta}(1, 1).$

Even though this model is conjugate, we will use pymc3 to perform inference with it for consistency with subsequent, non-conjugate models.

n_players = player_map.size
disadvantaged_id = df.disadvantaged_id.values

foul_called = df.foul_called.values
obs_rate = foul_called.mean()

with pm.Model() as bb_model:
    p = pm.Beta('p', 1., 1., shape=n_players)
    
    y = pm.Bernoulli('y_obs', p[disadvantaged_id],
                     observed=foul_called)

Throughout this post, we will use the no-U-turn sampler for inference, tuning the sampler’s hyperparameters for the first two thousand samples and subsequently keeping the next two thousand samples for inference.

N_TUNE = 2000
N_SAMPLES = 2000

SEED = 506421 # from random.org, for reproducibility

We now sample from the beta-Bernoulli model.

def sample(model, n_tune, n_samples, seed):
    with model:
        full_trace = pm.sample(n_tune + n_samples, tune=n_tune, random_seed=seed)
        
    return full_trace[n_tune:]

bb_trace = sample(bb_model, N_TUNE, N_SAMPLES, SEED)

Auto-assigning NUTS sampler...
Initializing NUTS using advi...
  2%|▏         | 4793/200000 [00:02<01:47, 1818.84it/s]Median ELBO converged.
Finished [100%]: Average ELBO = -3,554.3

100%|██████████| 4000/4000 [01:03<00:00, 63.38it/s]

We use energy energy plots to diagnose possible problems with our samples.

def energy_plot(trace):
    energy = trace['energy']
    energy_diff = np.diff(energy)
    
    fig, ax = plt.subplots(figsize=(8, 6))
    
    ax.hist(energy - energy.mean(), bins=30,
            lw=0, alpha=0.5,
            label="Energy")
    ax.hist(energy_diff, bins=30,
            lw=0, alpha=0.5,
            label="Energy difference")
    
    ax.set_xticks([])
    ax.set_yticks([])
    
    ax.legend()

energy_plot(bb_trace)

Since the energy and energy difference distributions are quite similar, there is no indication from this plot of sampling issues. For an in-depth treatment of Hamiltonian Monte Carlo algorithms and convergence diagnostics, consult Michael Betancourt’s excellent paper A Conceptual Introduction to Hamiltonian Monte Carlo.

We will use the widely applicable information criterion (WAIC) and binned residuals to check and compare our models. WAIC is a Bayesian measure of out-of-sample predictive accuracy based on in-sample data that is quite closely related to [leave-one-out cross-validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics%29#Leave-one-out_cross-validation). It attempts to improve upon known shortcomings of the widely-used deviance information criterion. (See Understanding predictive information criteria for Bayesian models for a review and comparison of various information criteria, including DIC and WAIC.) WAIC is easy to calculate with pymc3.

def get_waic_df(model, trace, name):
    with model:
        waic = pm.waic(trace)

    return pd.DataFrame.from_records([waic], index=[name], columns=waic._fields)

waic_df = get_waic_df(bb_model, bb_trace, "Beta-Bernoulli")

/opt/conda/lib/python3.5/site-packages/pymc3/stats.py:145: UserWarning: For one or more samples the posterior variance of the
        log predictive densities exceeds 0.4. This could be indication of
        WAIC starting to fail see http://arxiv.org/abs/1507.04544 for details
        
  """)

We see that the WAIC calculation indicates difficulties with the beta-Bernoulii model, which we will soon confirm.

waic_df

	WAIC	WAIC_se	p_WAIC
Beta-Bernoulli	6021.491064	66.23822	238.488377

In addition to the WAIC value, we get an estimate of its standard error (WAIC_se) and the number of effective parameters in the model (p_WAIC). The number of effective parameters is an indication of model complexity.

The second diagnostic tool we use on our models are binned residuals, which show how well-calibrated the model’s predicted probabilities are. Intuitively, if our model predicts that an event has a 35% chance of occurring and we can observe many repetitions of that event, we would expect the event to actually occur about 35% of the time. If the observed occurrences of the event differ substantially from the predicted rate, we have reason to doubt the quality of our model. Since we generally can’t observe each event many times, we instead group events into bins by their predicted probability and check that the average predicted probability in each bin is close to the rate at which the events in that bin are observed.

The binned predictions and residuals for the beta-Bernoulli model are shown below.

BINS = np.linspace(0, 1, 11)
BINS_3D = BINS[np.newaxis, np.newaxis]

def binned_residuals(y, p):
    p_3d = p[..., np.newaxis]

    in_bin = (BINS_3D[..., :-1] < p_3d) & (p_3d <= BINS_3D[..., 1:])
    bin_counts = in_bin.sum(axis=(0, 1))
    
    p_mean = (in_bin * p_3d).sum(axis=(0, 1)) / bin_counts
    y_mean = (in_bin * y[np.newaxis, :, np.newaxis]).sum(axis=(0, 1)) / bin_counts
    
    return y_mean, p_mean, bin_counts

def binned_residual_plot(bin_obs, bin_p, bin_counts):
    fig, (ax, resid_ax) = plt.subplots(ncols=2, sharex=True, figsize=(16, 6))

    ax.scatter(bin_p, bin_obs,
               s=300 * np.sqrt(bin_counts / bin_counts.sum()),
               zorder=5)
    ax.plot([0, 1], [0, 1], '--', c='k')

    ax.set_xlim(0, 1)
    ax.set_xticks(np.linspace(0, 1, 5))
    ax.xaxis.set_major_formatter(pct_formatter)
    ax.set_xlabel("Mean $p$ (binned)")

    ax.set_ylim(0, 1)
    ax.set_yticks(np.linspace(0, 1, 5))
    ax.yaxis.set_major_formatter(pct_formatter)
    ax.set_ylabel("Observed rate (binned)")

    resid_ax.scatter(bin_p, bin_obs - bin_p,
                     s=300 * np.sqrt(bin_counts / bin_counts.sum()),
                     zorder=5)

    resid_ax.hlines(0, 0, 1, 'k', '--')

    resid_ax.set_xlim(0, 1)
    resid_ax.set_xticks(np.linspace(0, 1, 5))
    resid_ax.xaxis.set_major_formatter(pct_formatter)
    resid_ax.set_xlabel("Mean $p$ (binned)")

    resid_ax.yaxis.set_major_formatter(pct_formatter)
    resid_ax.set_ylabel("Residual (binned)")

bin_obs, bin_p, bin_counts = binned_residuals(foul_called, bb_trace['p'][:, disadvantaged_id])

binned_residual_plot(bin_obs, bin_p, bin_counts)

In these plots, the dashed black lines show how these quantities would be related, for a perfect model. The area of each point is proportional to the number of events whose predicted probability fell in the relevant bin. From these plots, we get further confirmation that our simple beta-Bernoulli model is quite unsatisfactory, as many binned residuals exceed 5% in absolute value.

Below we plot the posterior mean and 90% credible interval for $p$ for each player in the data set (grouped by team, for legibility), along with the player’s observed foul called percentage when disadvantaged. The area of the point for observed foul called percentage is proportional to the number of plays in which the player was disadvantaged.

def to_param_df(player_df, trace, varnames):
    df = player_df
    
    for name in varnames:
        mean = trace[name].mean(axis=0)
        low, high = np.percentile(trace[name], [5, 95], axis=0)
        
        df = df.assign(**{
            '{}_mean'.format(name): mean[df.player_id],
            '{}_low'.format(name): low[df.player_id],
            '{}_high'.format(name): high[df.player_id]
        })
        
    return df

bb_df = to_param_df(team_player_map, bb_trace, ['p'])

def plot_params(mean, interval, names, ax=None, **kwargs):
    if ax is None:
        fig, ax = plt.subplots(figsize=(8, 6))
    
    n_players = names.size

    ax.errorbar(mean, np.arange(n_players),
                xerr=np.abs(mean - interval),
                fmt='o',
                label="Mean with\n90% interval")
    

    ax.set_ylim(-1, n_players)
    ax.set_yticks(np.arange(n_players))
    ax.set_yticklabels(names)
    
    return ax

def plot_p_params(rate, n_plays, league_mean, ax=None, **kwargs):
    if ax is None:
        ax = plt.gca()
        
    n_players = rate.size
        
    ax.scatter(rate, np.arange(n_players),
               c='k', s=20 * np.sqrt(n_plays),
               alpha=0.5, zorder=5,
               label="Observed")

    ax.vlines(league_mean, -1, n_players,
              'k', '--',
              label="League average")
    
def plot_p_helper(mean, low, high, rate, n_plays, names, league_mean=None, ax=None, **kwargs):
    if ax is None:
        ax = plt.gca()
    
    mean = mean.values
    rate = rate.values
    n_plays = n_plays.values
    names = names.values
    
    argsorted_ix = mean.argsort()
    interval = np.row_stack([low, high])
    
    plot_params(mean[argsorted_ix], interval[:, argsorted_ix], names[argsorted_ix],
                ax=ax, **kwargs)
    plot_p_params(rate[argsorted_ix], n_plays[argsorted_ix], league_mean,
                  ax=ax, **kwargs)

grid = sns.FacetGrid(bb_df, col='team', col_wrap=2,
                     sharey=False,
                     size=4, aspect=1.5)
grid.map(plot_p_helper,
         'p_mean', 'p_low', 'p_high',
         'disadvantaged_rate', 'disadvantaged_plays', 'name',
         league_mean=obs_rate);

grid.set_axis_labels(r"$p$", "Player");

for ax in grid.axes:
    ax.set_xticks(np.linspace(0, 1, 5));
    ax.set_xticklabels(ax.get_xticklabels(), visible=True)
    ax.xaxis.set_major_formatter(pct_formatter);
    
grid.fig.tight_layout();
grid.add_legend();

These plots reveal an undesirable property of this model and its inferences. Since the prior distribution on $p_i$ is uniform on the interval $[0, 1]$, all posterior estimates of $p_i$ are pulled towards the prior expected value of 50%. This phenomenon is known as shrinkage. In extreme cases of players that were never disadvantaged, the posterior estimate of $p_i$ is quite close to 50%. For these players, the league average foul call rate would seem to be a much more reasonable estimate of $p_i$ than 50%. The league average foul call rate is shown as a dotted black line on the charts above.

There are several possible modifications of the beta-Bernoulli model that can cause shrinkage toward the league average. Perhaps the most straightforward is the empirical Bayesian method that sets the parameters of the prior distribution on $p_i$ using the observed data. In this framework, there are many methods of choosing prior hyperparameters that make the prior expected value equal to the league average, therefore causing shrinkage toward the league average. We do not use empirical Bayesian methods in this post as they make it cumbersome to build the more complex models we want to use to understand the relationship between salary and foul calls. Empirical Bayesian methods are, however, an approximation to the fully hierachical models we begin building in the next section.

Hierarchical logistic-normal model

A hierarchical logistic-normal model addresses some of the shortcomings of the beta-Bernoulli model. For simplicity, this model focuses exclusively on the disadvantaged player and assumes that the log-odds of a foul call for a given disadvantaged player are normally distributed. That is,

\[ \begin{align*} \log \left(\frac{p_i}{1 - p_i}\right) & \sim N(\mu, \sigma^2) \\ \eta_{k} & = \log \left(\frac{p_{i(k)}}{1 - p_{i(k)}}\right), \end{align*}\]

which is equivalent to

\[p_{i(k)} = \frac{1}{1 + \exp(-\eta_k)}.\]

We address the beta-Bernoulli model’s shrinkage problem by placing a normal hyperprior distribution on $\mu$, $\mu \sim N(0, 100).$ This shared hyperprior makes this model hierarchical. To complete the specification of this model, we place a half-Cauchy prior on $\sigma$, $\sigma \sim \textrm{HalfCauchy}(2.5)$.

with pm.Model() as ln_model:
    μ = pm.Normal('μ', 0., 10.)
    Δ = pm.Normal('Δ', 0., 1., shape=n_players)
    σ = pm.HalfCauchy('σ', 2.5)
    
    p_player = pm.Deterministic('p_player', pm.math.sigmoid(μ + Δ * σ))

    η = μ + Δ[disadvantaged_id] * σ
    p = pm.Deterministic('p', pm.math.sigmoid(η))

    y = pm.Bernoulli('y_obs', p, observed=foul_called)

Throughout this post we use an offset parameterization for hierarchical models that significantly improves sampling efficiency. We now sample from this model.

ln_trace = sample(ln_model, N_TUNE, N_SAMPLES, SEED)

Auto-assigning NUTS sampler...
Initializing NUTS using advi...
  9%|▊         | 17001/200000 [00:12<02:13, 1372.87it/s]Median ELBO converged.
Finished [100%]: Average ELBO = -3,520.3

100%|██████████| 4000/4000 [02:27<00:00, 65.27it/s]

The energy plot for this model gives no cause for concern.

energy_plot(ln_trace)

We now calculate the WAIC of the logistic-normal model, and compare it to that of the beta-Bernoulli model.

waic_df = waic_df.append(get_waic_df(ln_model, ln_trace, "Logistic-normal"))

def waic_plot(waic_df):
    fig, (waic_ax, p_ax) = plt.subplots(ncols=2, sharex=True, figsize=(16, 6))

    waic_x = np.arange(waic_df.shape[0])

    waic_ax.errorbar(waic_x, waic_df.WAIC,
                     yerr=waic_df.WAIC_se,
                     fmt='o')

    waic_ax.set_xticks(waic_x)
    waic_ax.xaxis.grid(False)
    waic_ax.set_xticklabels(waic_df.index)
    waic_ax.set_xlabel("Model")

    waic_ax.set_ylabel("WAIC")

    p_ax.bar(waic_x, waic_df.p_WAIC)

    p_ax.xaxis.grid(False)
    p_ax.set_xlabel("Model")

    p_ax.set_ylabel("Effective number\nof parameters")

waic_plot(waic_df)

The left-hand plot above shows that the logistic-normal model is a significant improvement in WAIC over the beta-Bernoulli model, which is unsurprising. The right-hand plot shows that the logistic-normal model has roughly 20% the number of effective parameters as the beta-Bernoulli model. This reduction is due to the partial pooling effect of the hierarchical prior. The hyperprior on $\mu$ causes observations for one player to impact the estimate of $p_i$ for all players; this sharing of information across players is responsible for the large decrease in the number of effective parameters.

Finally, we examine the binned residuals for the logistic-normal model.

bin_obs, bin_p, bin_counts = binned_residuals(foul_called, ln_trace['p'])

binned_residual_plot(bin_obs, bin_p, bin_counts)

These binned residuals are much smaller than those of the beta-Bernoulli model, which is further confirmation that the logistic-normal model is preferable.

Below we plot the posterior distribution of $\operatorname{logit}^{-1}(\mu)$, and we see that the observed foul call rate of approximately 25.1% lies within its 90% interval.

ax, = pm.plot_posterior(ln_trace, ['μ'],
                        alpha_level=0.1, transform=expit, ref_val=obs_rate,
                        lw=0., alpha=0.75)

ax.xaxis.set_major_formatter(pct_formatter);

ax.set_title(r"$\operatorname{logit}^{-1}(\mu)$");

With this posterior for $\operatorname{logit}^{-1}(\mu)$, we see the desired posterior shrinkage of each $p_i$ toward the observed foul call rate.

ln_df = to_param_df(team_player_map, ln_trace, ['p'])

grid = sns.FacetGrid(ln_df, col='team', col_wrap=2,
                     sharey=False,
                     size=4, aspect=1.5)
grid.map(plot_p_helper,
         'p_mean', 'p_low', 'p_high',
         'disadvantaged_rate', 'disadvantaged_plays', 'name',
         league_mean=obs_rate);

grid.set_axis_labels(r"$p$", "Player");

for ax in grid.axes:
    ax.set_xticks(np.linspace(0, 1, 5));
    ax.set_xticklabels(ax.get_xticklabels(), visible=True)
    ax.xaxis.set_major_formatter(pct_formatter);
    
grid.fig.tight_layout();
grid.add_legend();

The inferences in these plots are markedly different from those of the beta-Bernoulli model. Most strikingly, we see that estimates have been shrunk towards the league average foul call rate, and that players that were never disadvantaged have posterior foul call probabilities quite close to that rate. As a consequence of this more reasonable shrinkage, the range of values taken by the posterior $p_i$ estimates is much smaller for the logistic-normal model than for the beta-Bernoulli model. Below we plot the top- and bottom-ten players by $p_i$.

fig, (top_ax, bottom_ax) = plt.subplots(nrows=2, sharex=True, figsize=(8, 10))

by_p = (ln_df.drop_duplicates(['player_id'])
             .sort_values('p_mean'))
p_top = by_p.iloc[-10:]

plot_params(p_top.p_mean.values, p_top[['p_low', 'p_high']].values.T,
            p_top.name.values,
            ax=top_ax);
top_ax.vlines(obs_rate, -1, 10,
              'k', '--',
              label=r"League average");

top_ax.xaxis.set_major_formatter(pct_formatter);

top_ax.set_ylabel("Player");

top_ax.set_title("Top ten");

p_bottom = by_p.iloc[:10]

plot_params(p_bottom.p_mean.values, p_bottom[['p_low', 'p_high']].values.T,
            p_bottom.name.values,
            ax=bottom_ax);
bottom_ax.vlines(obs_rate, -1, 10,
                 'k', '--',
                 label=r"League average");

bottom_ax.xaxis.set_major_formatter(pct_formatter);
bottom_ax.set_xlabel(r"$p$");

bottom_ax.set_ylabel("Player");

fig.tight_layout();
bottom_ax.legend(loc=1);
bottom_ax.set_title("Bottom ten");

Item-response (Rasch) model

The hierarchical logistic-normal model is certainly an improvement over the beta-Bernoulli model, but both of these models have focused solely on the disadvantaged player. It seems quite important to understand the contribution of not just the disadvantaged player, but also of the committing player in each play to the probability of a foul call. Item-response theory (IRT) provides generalizations the logistic-normal model that can account for the influence of both players involved in a play. IRT originated in psychometrics as a way to simultaneously measure individual aptitude and question difficulty based on test-response data, and has subsequently found many other applications. We use IRT to model foul calls by considering disadvantaged players as analagous to test takers and committing players as analagous to questions. Specifically, we will use the Rasch model for the probability $p_{i, j}$, that a foul is called on a play where player $i$ is disadvantaged by committing player $j$. This model posits that each player has a latent ability, $\theta_i$, that governs how often fouls are called when they are disadvantaged and a latent difficulty $b_j$ that governs how often fouls are not called when they are committing. The probability that a foul is called on a play where player $i$ is disadvantaged and player $j$ is committing is then a function of the difference between the corresponding latent ability and difficulty parameters,

\[ \begin{align*} \eta_k & = \theta_{i(k)} - b_{j(k)} \\ p_k & = \frac{1}{1 + \exp(-\eta_k)}. \end{align*} \]

In this model, a player with a large value of $\theta_i$ is more likely to get a foul called when they are disadvantaged, and a player with a large value of $b_j$ is less likely to have a foul called when they are committing. If $\theta_{i(k)} = b_{j(k)}$, there is a 50% chance a foul is called on that play.

To complete the specification of this model, we place priors on $\theta_i$ and $b_j$. Similarly to $\eta$ in the logistic-normal model, we place a hierarchical normal prior on $\theta_i$,

\[ \begin{align*} \mu_{\theta} & \sim N(0, 100) \\ \sigma_{\theta} & \sim \textrm{HalfCauchy}(2.5) \\ \theta_i & \sim N(\mu_{\theta}, \sigma^2_{\theta}). \end{align*} \]

with pm.Model() as rasch_model:
    μ_θ = pm.Normal('μ_θ', 0., 10.)
    Δ_θ = pm.Normal('Δ_θ', 0., 1., shape=n_players)
    σ_θ = pm.HalfCauchy('σ_θ', 2.5)
    θ = pm.Deterministic('θ', μ_θ + Δ_θ * σ_θ)

We also place a hierarchical normal prior on $b_j$, though this prior must be subtley different from that on $\theta_i$. Since $\theta_i$ and $b_j$ are latent variables, there is no natural scale on which they should be measured. If each $\theta_i$ and $b_j$ are shifted by the same amount, say $\delta$, the likelihood does not change. That is, if $\tilde{\theta}_i = \theta_i + \delta$ and $\tilde{b}_j = b_j + \delta$, then

\[ \tilde{\eta}_{i, j} = \tilde{\theta}_i - \tilde{b}_j = \theta_i + \delta - (b_j + \delta) = \theta_i - b_j = \eta_{i, j}. \]

Therefore, if we allow $\theta_i$ and $\beta_j$ to be shifted by arbitrary amounts, the Rasch model is not identified. We identify the Rasch model by constraining the mean of the hyperprior on $b_j$ to be zero,

\[ \begin{align*} \sigma_b & \sim \textrm{HalfCauchy}(2.5) \\ b_j & \sim N(0, \sigma^2_b). \end{align*} \]

with rasch_model:
    Δ_b = pm.Normal('Δ_b', 0., 1., shape=n_players)
    σ_b = pm.HalfCauchy('σ_b', 2.5)
    b = pm.Deterministic('b', Δ_b * σ_b)

We now specify the Rasch model’s likelihood and sample from it.

committing_id = df.committing_id.values

with rasch_model:
    η = θ[disadvantaged_id] - b[committing_id]
    p = pm.Deterministic('p', pm.math.sigmoid(η))

    y = pm.Bernoulli('y_obs', p, observed=foul_called)

rasch_trace = sample(rasch_model, N_TUNE, N_SAMPLES, SEED)

Auto-assigning NUTS sampler...
Initializing NUTS using advi...
Average ELBO = -3,729.5:  11%|█▏        | 22583/200000 [00:19<02:33, 1156.17it/s]Median ELBO converged.
Finished [100%]: Average ELBO = -3,037.1

100%|██████████| 4000/4000 [02:01<00:00, 32.81it/s]

Again, the energy plot for this model gives no cause for concern.

energy_plot(rasch_trace)

Below we show the WAIC of our three models.

waic_df = waic_df.append(get_waic_df(rasch_model, rasch_trace, "Rasch"))

waic_plot(waic_df)

The Rasch model represents a moderate WAIC improvement over the logistic-normal model, and unsurprisingly has many more effective parameters (since it added a nominal parameter, $b_j$, per player).

The Rasch model also has reasonable binned residuals, with very few events having residuals above 5%.

bin_obs, bin_p, bin_counts = binned_residuals(foul_called, rasch_trace['p'])

binned_residual_plot(bin_obs, bin_p, bin_counts)

For the Rasch model (and subsequent models), we switch from visualizing the per-player call probabilities to the latent parameters $\theta_i$ and $b_j$.

μ_θ_mean = rasch_trace['μ_θ'].mean()

rasch_df = to_param_df(team_player_map, rasch_trace, ['θ', 'b'])

def plot_params_helper(mean, low, high, names, league_mean=None, league_mean_name=None, ax=None, **kwargs):
    if ax is None:
        ax = plt.gca()
    
    mean = mean.values
    names = names.values
    
    argsorted_ix = mean.argsort()
    interval = np.row_stack([low, high])
    
    plot_params(mean[argsorted_ix], interval[:, argsorted_ix], names[argsorted_ix],
                ax=ax, **kwargs)
    
    if league_mean is not None:
        ax.vlines(league_mean, -1, names.size,
                  'k', '--',
                  label=league_mean_name)

grid = sns.FacetGrid(rasch_df, col='team', col_wrap=2,
                     sharey=False,
                     size=4, aspect=1.5)
grid.map(plot_params_helper,
         'θ_mean', 'θ_low', 'θ_high', 'name',
         league_mean=μ_θ_mean,
         league_mean_name=r"$\mu_{\theta}$");

grid.set_axis_labels(r"$\theta$", "Player");

grid.fig.tight_layout();
grid.add_legend();

grid = sns.FacetGrid(rasch_df, col='team', col_wrap=2,
                     sharey=False,
                     size=4, aspect=1.5)
grid.map(plot_params_helper,
         'b_mean', 'b_low', 'b_high', 'name',
         league_mean=0.);

grid.set_axis_labels(r"$b$", "Player");

grid.fig.tight_layout();
grid.add_legend();

Though these plots are voluminuous and therefore difficult to interpret precisely, a few trends are evident. The first is that there is more variation in the committing skill ($b_j$) than in disadvantaged skill ($\theta_i$). This difference is confirmed in the following histograms of the posterior expected values of $\theta_i$ and $b_j$.

def plot_latent_distributions(θ, b):
    fig, (θ_ax, b_ax) = plt.subplots(nrows=2, sharex=True, figsize=(8, 6))
    
    bins = np.linspace(0.9 * min(θ.min(), b.min()),
                       1.1 * max(θ.max(), b.max()),
                       75)

    θ_ax.hist(θ, bins=bins,
              alpha=0.75)

    θ_ax.xaxis.set_label_position('top') 
    θ_ax.set_xlabel(r"Posterior expected $\theta$")

    θ_ax.set_yticks([])
    θ_ax.set_ylabel("Frequency")

    b_ax.hist(b, bins=bins,
              color=green, alpha=0.75)

    b_ax.xaxis.tick_top()
    b_ax.set_xlabel(r"Posterior expected $b$")

    b_ax.set_yticks([])
    b_ax.invert_yaxis()
    b_ax.set_ylabel("Frequency")

    fig.tight_layout()

plot_latent_distributions(rasch_df.θ_mean, rasch_df.b_mean)

The following plots show the top and bottom ten players in terms of both $\theta_i$ and $b_j$.

def top_10_plot(trace_df, μ_θ=0):
    fig = plt.figure(figsize=(16, 10))
    θ_top_ax = fig.add_subplot(221)
    b_top_ax = fig.add_subplot(222)
    θ_bottom_ax = fig.add_subplot(223, sharex=θ_top_ax)
    b_bottom_ax = fig.add_subplot(224, sharex=b_top_ax)

    # necessary for players that have been on more than one team
    trace_df = trace_df.drop_duplicates(['player_id'])
    
    by_θ = trace_df.sort_values('θ_mean')
    θ_top = by_θ.iloc[-10:]
    θ_bottom = by_θ.iloc[:10]

    plot_params(θ_top.θ_mean.values, θ_top[['θ_low', 'θ_high']].values.T,
                θ_top.name.values,
                ax=θ_top_ax)
    θ_top_ax.vlines(μ_θ, -1, 10,
                    'k', '--',
                    label=(r"$\mu_{\theta}$" if μ_θ != 0 else "League average"))

    plt.setp(θ_top_ax.get_xticklabels(), visible=False)
    
    θ_top_ax.set_ylabel("Player")

    θ_top_ax.legend(loc=2)
    θ_top_ax.set_title("Top ten")

    plot_params(θ_bottom.θ_mean.values, θ_bottom[['θ_low', 'θ_high']].values.T,
                θ_bottom.name.values,
                ax=θ_bottom_ax)
    θ_bottom_ax.vlines(μ_θ, -1, 10,
                       'k', '--',
                       label=(r"$\mu_{\theta}$" if μ_θ != 0 else "League average"))

    θ_bottom_ax.set_xlabel(r"$\theta$")

    θ_bottom_ax.set_ylabel("Player")

    θ_bottom_ax.set_title("Bottom ten")

    by_b = trace_df.sort_values('b_mean')
    b_top = by_b.iloc[-10:]
    b_bottom = by_b.iloc[:10]

    plot_params(b_top.b_mean.values, b_top[['b_low', 'b_high']].values.T,
                b_top.name.values,
                ax=b_top_ax)
    b_top_ax.vlines(0, -1, 10,
                    'k', '--',
                    label="League average");

    plt.setp(b_top_ax.get_xticklabels(), visible=False)

    b_top_ax.legend(loc=2)
    b_top_ax.set_title("Top ten")

    b_bottom_player_id = b.argsort()[:10]

    plot_params(b_bottom.b_mean.values, b_bottom[['b_low', 'b_high']].values.T,
                b_bottom.name.values,
                ax=b_bottom_ax)
    b_bottom_ax.vlines(0, -1, 10,
                       'k', '--')

    b_bottom_ax.set_xlabel(r"$b$")

    b_bottom_ax.set_title("Bottom ten")

    fig.tight_layout()

top_10_plot(rasch_df, μ_θ=μ_θ_mean)

We focus first on $\theta_i$. Interestingly, the top-ten players for the Rasch model contains many more top-tier stars than the logistic-normal model, including John Wall, Russell Westbrook, and LeBron James. Turning to $b$, it is interesting that the while the top and bottom ten players contain many recognizable names (LaMarcus Aldridge, Harrison Barnes, Kawhi Leonard, and Ricky Rubio) the only truly top-tier player present is Anthony Davis.

Time remaining model

As basketball fans know, there are many factors other than the players involved that influence foul calls. Very often, sufficiently close NBA games end with intentional fouls, as the losing team attempts to stop the clock and force another offensive possesion. Therefore, we expect to see in increase in the foul call probability as the game nears its conclusion.

n_sec = 121
sec = (df.seconds_left
         .round(0)
         .values
         .astype(np.int64))

fig, ax = plt.subplots(figsize=(8, 6))

(df.groupby(sec)
   .foul_called
   .mean()
   .plot(c='k', label="Observed foul call rate", ax=ax));

ax.set_xticks(np.linspace(0, 120, 5));
ax.invert_xaxis();
ax.set_xlabel("Seconds remaining in game");

ax.yaxis.set_major_formatter(pct_formatter);
ax.set_ylabel("Probability foul is called");

ax.legend(loc=2);

The plot above confirms this expectation, which we can use to improve our latent skill model. If $t \in \{0, 1, \ldots, 120\}$ is the number of seconds remaining in the game, we model the latent contribution of $t$ to the logodds that a foul is called with a Gaussian random walk,

\[ \begin{align*} \lambda_0 & \sim N(0, 100) \\ \lambda_t & \sim N(\lambda_{t - 1}, \tau^{-1}_{\lambda}) \\ \tau_{\lambda} & \sim \textrm{Exp}(10^{-4}). \end{align*} \]

This prior allows us to flexibly model the shape of the curve shown above. If $t(k)$ is the number of seconds remaining during the $k$-th play, we incorporate $\lambda_{t(k)}$ into our model with

\[\eta_k = \lambda_{t(k)} + \theta_{i(k)} - b_{j(k)}.\]

This model is not identified until we constrain the mean of $\theta$ to be zero, for reasons similar to those discussed above for the Rasch model.

with pm.Model() as time_model:
    τ_λ = pm.Exponential('τ_λ', 1e-4)
    λ = pm.GaussianRandomWalk('λ', tau=τ_λ,
                              init=pm.Normal.dist(0., 10.),
                              shape=n_sec)
    
    Δ_θ = pm.Normal('Δ_θ', 0., 1., shape=n_players)
    σ_θ = pm.HalfCauchy('σ_θ', 2.5)
    θ = pm.Deterministic('θ', Δ_θ * σ_θ)
    
    Δ_b = pm.Normal('Δ_b', 0., 1., shape=n_players)
    σ_b = pm.HalfCauchy('σ_b', 2.5)
    b = pm.Deterministic('b', Δ_b * σ_b)
    
    η = λ[sec] + θ[disadvantaged_id] - b[committing_id]
    p = pm.Deterministic('p', pm.math.sigmoid(η))

    y = pm.Bernoulli('y_obs', p, observed=foul_called)

We now sample from the model.

time_trace = sample(time_model, N_TUNE, N_SAMPLES, SEED)

Auto-assigning NUTS sampler...
Initializing NUTS using advi...
Average ELBO = -1.0533e+05:  15%|█▌        | 30300/200000 [00:31<02:52, 982.03it/s] Median ELBO converged.
Finished [100%]: Average ELBO = -3,104.5

100%|██████████| 4000/4000 [03:10<00:00, 20.99it/s]

The energy plot for this model is worse than the previous ones, but not too bad.

energy_plot(time_trace)

waic_df = waic_df.append(get_waic_df(time_model, time_trace, "Time"))

waic_plot(waic_df)

We see that the time remaining model represents an appreciable improvement over the Rasch model in terms of WAIC.

bin_obs, bin_p, bin_counts = binned_residuals(foul_called, time_trace['p'])

binned_residual_plot(bin_obs, bin_p, bin_counts)

The binned residuals for this model also look quite good, with very few samples appreciably exceeding a 1% difference.

We now compare the distribtuions of $\theta_i$ and $b_j$ for this model with those for the Rasch model.

time_df = to_param_df(team_player_map, time_trace, ['θ', 'b'])

plot_latent_distributions(time_df.θ_mean, time_df.b_mean)

The effect of constraining the mean of $\theta$ to be zero is immediately apparent. Also, the variation in $\theta$ remains lower than the variation than $b$ in this model. We also see that the top- and bottom-ten players by $\theta$- and $b$-value remain largely unchanged from the Rasch model.

top_10_plot(time_df)

Basketball fans may find it amusing that under this model, Dwight Howard has joined the top-ten in terms of $\theta$ and Ricky Rubio is no longer the worst player in terms of $b$.

While this model has not done much to change the rank-ordering of the most- and least-skilled players, it does enable us to plot per-player foul probabilities over time, as below.

fig, (θ_ax, b_ax) = plt.subplots(ncols=2, sharex=True, sharey=True, figsize=(16, 6))

(df.groupby(sec)
   .foul_called
   .mean()
   .plot(c='k', alpha=0.5,
         label="Observed foul call rate",
         ax=θ_ax));

plot_sec = np.arange(n_sec)

θ_ax.plot(plot_sec, expit(time_trace['λ'].mean(axis=0)),
          c='k',
          label="Average player");

θ_best_id = time_df.ix[time_df.θ_mean.idxmax()].player_id
θ_ax.plot(plot_sec, expit(time_trace['θ'][:, θ_best_id].mean(axis=0) \
                          + time_trace['λ'].mean(axis=0)),
          label=player_map[θ_best_id]);

θ_worst_id = time_df.ix[time_df.θ_mean.idxmin()].player_id
θ_ax.plot(plot_sec, expit(time_trace['θ'][:, θ_worst_id].mean(axis=0) \
                          + time_trace['λ'].mean(axis=0)),
          label=player_map[θ_worst_id]);


θ_ax.set_xticks(np.linspace(0, 120, 5));
θ_ax.invert_xaxis();
θ_ax.set_xlabel("Seconds remaining in game");

θ_ax.yaxis.set_major_formatter(pct_formatter);
θ_ax.set_ylabel("Probability foul is called\nagainst average opposing player");

θ_ax.legend(loc=2);
θ_ax.set_title(r"Disadvantaged player ($\theta$)");

(df.groupby(sec)
   .foul_called
   .mean()
   .plot(c='k', alpha=0.5,
         label="Observed foul call rate",
         ax=b_ax));

plot_sec = np.arange(n_sec)

b_ax.plot(plot_sec, expit(time_trace['λ'].mean(axis=0)),
          c='k',
          label="Average player");

b_best_id = time_df.ix[time_df.b_mean.idxmax()].player_id
b_ax.plot(plot_sec, expit(-time_trace['b'][:, b_best_id].mean(axis=0) \
                          + time_trace['λ'].mean(axis=0)),
          label=player_map[b_best_id]);

b_worst_id = time_df.ix[time_df.b_mean.idxmin()].player_id
b_ax.plot(plot_sec, expit(-time_trace['b'][:, b_worst_id].mean(axis=0) \
                          + time_trace['λ'].mean(axis=0)),
          label=player_map[b_worst_id]);


b_ax.set_xticks(np.linspace(0, 120, 5));
b_ax.invert_xaxis();
b_ax.set_xlabel("Seconds remaining in game");

b_ax.legend(loc=2);
b_ax.set_title(r"Committing player ($b$)");

Here we have plotted the probability of a foul call while being opposed by an average player (for both $\theta$ and $b$), along with the probability curves for the players with the highest and lowest $\theta$ and $b$, respectively. While these plots are quite interesting, one weakness of our model is that the difference between each player’s curve and the league average is constant over time. It would be an interesting and useful to extend this model to allow player offsets to vary over time. Additonally, it would be interesting to understand the influence of the score on the foul-called rate as the game nears its end. It seems quite likely that the winning team is much less likely to commit fouls while the losing team is much more likely to to commit intentional fouls in close games.

We now plot the per-player values of $\theta_i$ and $b_j$ under this model.

grid = sns.FacetGrid(time_df, col='team', col_wrap=2,
                     sharey=False,
                     size=4, aspect=1.5)
grid.map(plot_params_helper,
         'θ_mean', 'θ_low', 'θ_high', 'name',
         league_mean=0.,
         league_mean_name="League average");

grid.set_axis_labels(r"$\theta$", "Player");

grid.fig.tight_layout();
grid.add_legend();

grid = sns.FacetGrid(time_df, col='team', col_wrap=2,
                     sharey=False,
                     size=4, aspect=1.5)
grid.map(plot_params_helper,
         'b_mean', 'b_low', 'b_high', 'name',
         league_mean=0.,
         league_mean_name="League average");

grid.set_axis_labels(r"$b$", "Player");

grid.fig.tight_layout();
grid.add_legend();

Salary model

Our final model uses salary as a proxy for “star power” to explore its influence on foul calls. We also (somewhat naively) impute missing salaries to the (log) league average.

std_log_salary = (salary_df.ix[np.arange(n_players)]
                           .std_log_salary
                           .fillna(0)
                           .values)

With $s_i$ denoting the $i$-the player’s standardized log salary, our model becomes

\[ \begin{align*} \theta_i & = \theta_{0, i} + \delta_{\theta} \cdot s_i \\ b_j & = b_{0, j} + \delta_b \cdot s_j \\ \eta_k & = \lambda_{t(k)} + \theta_{i(k)} - b_{j(k)}. \end{align*} \]

In this model, each player’s $\theta_i$ and $b_j$ parameters are linear functions of their standardized log salary, with (hierarchical) varying intercepts. The varying intercepts $\theta_{0, i}$ and $b_{0, j}$ are endowed with the same hierarchical normal priors as $\theta_i$ and $b_j$ had in the previous model. We place normal priors, $\delta_{\theta} \sim N(0, 100)$ and $\delta_b \sim N(0, 100)$, on the salary coefficients.

with pm.Model() as salary_model:
    τ_λ = pm.Exponential('τ_λ', 1e-4)
    λ = pm.GaussianRandomWalk('λ', tau=τ_λ,
                              init=pm.Normal.dist(0., 10.),
                              shape=n_sec)
    
    Δ_θ0 = pm.Normal('Δ_θ0', 0., 1., shape=n_players)
    σ_θ0 = pm.HalfCauchy('σ_θ0', 2.5)
    θ0 = pm.Deterministic('θ0', Δ_θ0 * σ_θ0)
    
    δ_θ = pm.Normal('δ_θ', 0., 10.)
    
    θ = pm.Deterministic('θ', θ0 + δ_θ * std_log_salary)
    
    Δ_b0 = pm.Normal('Δ_b0', 0., 1., shape=n_players)
    σ_b0 = pm.HalfCauchy('σ_b0', 2.5)
    b0 = pm.Deterministic('b0', Δ_b0 * σ_b0)
    
    δ_b = pm.Normal('δ_b', 0., 10.)
    
    b = pm.Deterministic('b', b0 + δ_b * std_log_salary)
    
    η = λ[sec] + θ[disadvantaged_id] - b[committing_id]
    p = pm.Deterministic('p', pm.math.sigmoid(η))

    y = pm.Bernoulli('y_obs', p, observed=foul_called)

salary_trace = sample(salary_model, N_TUNE, N_SAMPLES, SEED)

Auto-assigning NUTS sampler...
Initializing NUTS using advi...
Average ELBO = -1.0244e+05:  15%|█▌        | 30998/200000 [00:32<02:57, 952.52it/s]Median ELBO converged.
Finished [100%]: Average ELBO = -2,995.3

100%|██████████| 4000/4000 [03:45<00:00, 17.76it/s]

The energy plot for this model looks a bit worse than that for the time remaining model.

energy_plot(salary_trace)

The salary model also appears to be a slight improvement over the time remaining model in terms of WAIC.

waic_df = waic_df.append(get_waic_df(salary_model, salary_trace, "Salary"))

waic_plot(waic_df)

The binned residuals continue to look good for this model.

bin_obs, bin_p, bin_counts = binned_residuals(foul_called, salary_trace['p'])

binned_residual_plot(bin_obs, bin_p, bin_counts)

Based on the posterior distributions of $\delta_{\theta}$ and $\delta_b$, we expect to see a fairly strong relationship between a player’s (standardized log) salary and their latent skill parameters.

pm.plot_posterior(salary_trace, ['δ_θ', 'δ_b'],
                  lw=0., alpha=0.75);

The following plots confirm this relationship.

salary_df_ = to_param_df(team_player_map, salary_trace, ['θ', 'θ0', 'b', 'b0'])

fig, (θ_ax, b_ax) = plt.subplots(ncols=2, sharex=True, figsize=(16, 6))

salary = (salary_df.ix[np.arange(n_players)]
                   .salary
                   .fillna(salary_df.salary.mean())
                   .values)

θ_ax.scatter(salary[salary_df_.player_id], salary_df_.θ_mean,
             alpha=0.75);

θ_ax.xaxis.set_major_formatter(million_dollars_formatter);
θ_ax.set_xlabel("Salary");

θ_ax.set_ylabel(r"$\theta$");

b_ax.scatter(salary[salary_df_.player_id], salary_df_.b_mean,
             alpha=0.75);

b_ax.xaxis.set_major_formatter(million_dollars_formatter);
b_ax.set_xlabel("Salary");

b_ax.set_ylabel(r"$b$");

It is important to note here that these relationships are descriptive, not causal. Our original intent was to use salary as a proxy for the difficult-to-quantify notion of “star power.” These plots suggest that the probability of getting a foul called when disadvanted and not called when committing are both positively related to salary, and therefore (a bit more dubiously) star power.

Importantly, $\theta_i$ and $b_j$ should no longer be interpreted directly as measuring latent skill, which is presumably intrinsic to a player and not directly dependent on their salary. In fact, it seems at plausible that NBA scouts, front offices, players, and agents would be somewhat able to appreciate these latent skills, place a value on them, and thereby price them into contracts. It would be interesting future work to refine this model to give an econometric answer to this question.

Since we shouldn’t interpret $\theta_i$ and $b_j$ as measures of latent skill in this model, we will not plot their per-player distributions.

Conclusion

We set out to explore the relationship between players involved in a play and the probability that a foul is called, along with other factors related to that probability. Through a series of progressively more complex Bayesian item-response models, we have seen that

foul call probability does vary appreciably across both disadvantaged and committing player,
there is more variation in the latent skill of the committing player to avoid a foul call than there is in the variation in the latent skill of the disadvantaged player to draw a foul call,
the amount of time remaining in the game is strongly related to the probability of a foul call, and
there is a positive relationship between player salary and the probability that a foul is called when they are disadvantaged and not called when they are committing. With a bit of a leap, we can say that the probability a foul is called is at least loosely related to the “star power” of the players involved.

In this post we have only scratched the surface of Bayesian item-response theory. For a more in-depth treatment, consult Bayesian Item Response Modeling.

This is is available as a Jupyter notebook here.