What's the matter with Win Probability?

Looks like I got this post up just in time. Brian has introduced a series of updates to the Win Probability calculator that largely address the WP overconfidence noted here. He also added a way to adjust the model for pregame expectations of team strength.

Brian Burke’s Win Probability (WP) stat (see here for the explanation and here for the weekly game visualizations) is a must for fans of NFL analytics. The individual game visualizations might as well be a box score because they track the ebb and flow of a game with far greater detail than quarterly score totals or drive descriptions in about the same amount of real estate.

Watching the WP metrics this season, however, I can’t help but feel that there are more wild swings than in previous years. After Detroit’s comeback against the Cowboys this weekend I decided I needed to take a look – despite having been burned by my biases on previous hunts initiated by anecdotal evidence.

WP Background

The stat is created by looking at previous seasons and evaluating whether teams in similar circumstances (down & distance, field position, time left, score differential) ended up winning their game. The resulting stat is not so much a forward-looking projection as a proportion of the teams that ended up winning from that state. Herein lies the potential problem: if the “previous seasons” in question stretch back too far the game may have been different enough that a 7 point lead doesn’t mean the same thing while if the “previous seasons” in question doesn’t go back far enough there will not be enough data for the myriad configurations of the variables.
 
Stats

Using my trusty, homemade Monte Carlo simulator I’ll start by looking at the number of teams to win after being the first in their game to reach a WP of 90% (WP90).

In the 120 games played through Week 8 of the NFL season, 106 of them have ended with the first team to reach WP90 as the winner[1]. This won’t account for where a team got to above WP90 before losing but does give us a nice, conservative place to start.

Looking at 1000 simulations of the 120 games this season there is an average WP90 games won at 108 (go figure) with a sample StDev of 3.2. Of the simulated seasons, 10.3% ended with 106 WP90 winners and 18.5% ended with even fewer. This implies the chance of seeing 106 out of 120 is roughly 28.8% - slightly uncommon but not really rare.

Digging into the data a bit further there is an interesting revelation. The 14 teams – actually 12 with Tampa Bay and Dallas both appearing on the list twice – that went on to lose after reaching WP90 actually tended to go well beyond WP90 before losing. Four of the teams reached WP99 and two reached WP98. If we are generous and give them all WP98, we would still only expect a 3.4% chance of seeing that many losses according to the simulator. If we are not generous and give them WP98.5, we only see about 1% chance of that many losses.

Analysis

There’s something going on at the far end of Win Probability and I suspect a combination of increased pace of offenses, more efficient offenses and out-of-sample results.

• Increased Pace – I wrote about this last week but it goes beyond the overall pace in a neutral situation. With pace being a focus this offseason after New England's success and Chip Kelly's hiring, many more teams than usual came in armed with a no huddle offense that can be quite handy during two minute drills or against big fourth quarter deficits
• More Efficient Offenses – Regardless of how far back Brian’s data goes this season is out of sample so far in terms of offensive performance compared with 1, 5 or 10 years ago. 23.1 points per game compares with 22.8 in 2012, 22.0 in 2008 and 20.8 in 2003. Plays per game are only up 5% in the same period while pass yards per game are up over 20%, favoring comebacks from big deficits.
• Out-of-Sample Results – Some coaches appear to be very creative at coming up with new ways to win or lose (especially lose, very creative there) games. Until a game has been blown in a certain way, and really until a game has been blown from a specific point on the field with a specific amount of time left, the model won’t acknowledge the possibility that it can happen.

For all that, Win Probability remains my preferred way to evaluate in-game decisions and catch up quickly on a game I missed. As the league finds equilibrium in the offense/defense continuum these results should become more in line with the stated percentages.  

[1] This saves me from having to do any quirky math to account for the fact that 2 teams (New Orleans in Week 2 against Tampa Bay and Cincinnati in Week 3 against Green Bay) were first to WP90 before dropping below WP10 and then rallying to win the game. Happily enough, if we consider that there are 16 teams (the 14 losers plus Cincy and New Orleans) to go above WP90 and below WP10 in the same game, it is reasonable to believe that roughly 10% of them would recover to win.

Faster 3 & outs for everyone: Pace of play in the NFL

One of the biggest stories of the NFL offseason was the impending rollout of fast-paced offenses that would limit time between plays – and substitutions – in the name of putting the defense on its heels and racking up points like a college team. With Chip Kelly coming into the league bringing his high-flying Oregon offense and teams across the league finding success with no huddle offense the time seemed right for this style to sweep the league. Former NFL coach Nick Saban (I’m told he’s in a position of some note in college football) waded into the fray to suggest that fast-paced offenses might cause more injuries than traditional, pro-style offenses (worth looking at but not in this post).

Now that we’re more than a quarter of the way into the season it is time to check the record and see how much the game has really changed from year to year. Are fast-paced teams the vanguard of a new way to play NFL football – racking up yards on the scale of the Oregon Ducks or the early-90s Bills – or are they copycats blindly running through quicker 3 & outs because they aren’t good enough to copy those Ducks and last season’s fast-paced Patriots offense that led the league in speed and offensive efficiency?

Pace 

First of all, we need to see if teams are actually speeding up relative to past years. Football Outsiders has a pace stat that excludes plays where the situation dictates the pace to look at how a team plays when they can do whatever they want that we’ll use to examine if the league is really changing[1]. As you would expect, in 2013 so far the biggest gaps between situation-neutral are for winless Jacksonville (4.7 seconds faster overall), the winless New York Giants (4.6 seconds) and the only-slighly-more-win-possessing Washington (4.3 seconds) as the score dictates that they play much more quickly than they would otherwise prefer.

The median team’s situation-neutral play from 2008 to 2013 is shown below.



The results are even more striking if we look at the individual teams. The figure below shows teams ranked from fastest to slowest for each season. 

NFL Spending on RBs

For those of you who really wanted to read through the whole post I did on the value of running backs but just didn't make it, here's the Tableau visualization from the end. Check both tabs to see the percent of team cap spend on running backs and then all players with running backs highlighted.

Learn About Tableau

Nobody spends money on running backs, right?

In the wake of the Trent Richardson trade from Cleveland to Indianapolis, it seems like a lot of conventional wisdom holds that running backs are overpaid and teams are stupid to pay them a lot of money. Please enjoy some out of context tweets that support my assertion:

This Browns regime didn't draft Richardson, it's an era where fewer RBs going 1st round and if they don't value him that highly, makes sense
— Jason La Canfora (@JasonLaCanfora) September 18, 2013

also, on this Richardson deal, this had nothing to do with "scheme" or "fit." it was about value and upside evaluation on the Browns part
— Jason La Canfora (@JasonLaCanfora) September 19, 2013

Browns source says Richardson just wasn't a great fit, so getting a 1 was great value. Cleveland now has 2 1's, a 2, 2 3's and 2 4's in '14.
— Albert Breer (@AlbertBreer) September 18, 2013

Good move by Browns to get what they could for a player drafted too high.
— Steve Drake (@SteveDrake_) September 18, 2013

Ask yourself: if Trent Richardson were a 4th round pick, what would Indy have traded for him?
— Steve Drake (@SteveDrake_) September 18, 2013

Nervous that Cleveland, the team I grew up with except for those three years when they were gone and the subsequent years when they’ve been terrible, had made a mistake, I wanted to dig into the data and see about the “devaluation” of running backs in practice. Unfortunately for the timeliness of this post I did not have the data handy and had to construct the data set.

Working from a number of sources I duct taped together a data set that could serve my needs, tied name/team/year combinations to positions and stepped back to look at the result.

This does not fit my narrative. Fear not, though, because we haven’t yet taken into account the contemporary increase in the salary cap – so substantial in the 2003 to 2009 period.