Revisiting Replacement Officials in the NFL

The go-to story for sportswriters in the first few weeks of the season was the use of replacement officials in the NFL. In the first week, when things appeared to be going well, a few of them trotted out the “they’re not so bad” line. By week two, however, there was solid consensus around the fact that things needed to change. After some further shenanigans in week three the NFL came around and settled with the real officials.

A couple weeks ago I took a look at how the replacements did. The conclusion: they were pretty much as bad as people said. They didn’t call a ton of extra penalties, but the increase was concentrated in some high-impact calls: Offensive Holding, Defensive Pass Interference, Defensive Holding and Personal Fouls. What better way to return from vacation than to circle back on this now that we have several weeks each of the replacements and the real officials.

Lots of talk about numbers…

The task of comparing the two sets of officials is complicated by a couple of additional factors. The number of games played in a given week varies as teams have byes. This is a pretty simple one as we just have to pull in the actual number of games played and convert the penalties into per-game numbers.

The tougher one is the overall trend in penalties throughout the season. Teams commit different penalties at different rates in the beginning of the season as opposed to the middle. To address this we will normalize the replacements’ statistics from weeks 1-3 with the averages of the past ten years in weeks 1-3 and normalize the real officials’ statistics from weeks 4-7 with the averages of the past ten years in weeks 4-7. Seeing that both, for example, are at 110 for False Starts does not mean that the numbers are the same – it means that both the replacements and real officials were calling False Starts at a rate 10% higher than the average of the past ten years for their respective weeks.

The part where we actually see if things have changed

The first thing that jumps out of the data is that each of the four penalties noted above has dropped significantly now that the real officials are back. Those calls that seemed relatively straightforward such as False Starts are unchanged from replacements to real officials. The penalties that were causing the game to feel wrong, the judgment calls for which the replacements did not have the right benchmark, have come down considerably. Personal Fouls have been cut in half, relative to their respective ten year averages, from the replacements to the real officials.

Interestingly, the drop in Defensive Pass Interference was accompanied by an increase in calls of Offensive Pass Interference. The raw number increase, 0.20, is fairly similar to the decrease, -0.29, though the percentage change looks more impressive for Offensive Pass Interference because the graph is ordered from most frequently called to least. One can imagine several scenarios to explain this. Perhaps the offensive players got used to getting calls on contested balls and started playing sloppier. Alternatively, the real officials may have been told by the NFL to get aggressive on pass interference to bring things back to normal.

A closer look at the stats has convinced me more than ever that the replacements had to go. If the NFL wants to try this strategy in the future they need to have a contingency plan in place for actually, you know, officiating the games.