Luck vs. Skill - Year over Year NFL Draft Performance

Note - I've gone back to this issue a couple times since this post and looked at whether there is any evidence for skill in sequential picks in a single draft and in picks made after trades up. Check them out via the embedded links.

Both Brian Burke and Chase Stuart recently wrote on the phenomenon of drafting in the NFL and the randomness that goes with it. Chase in particular wrote about the question of whether some teams are better than others at drafting.

At the recommendation of my friend Sarah, I recently read Michael Mauboussin's book The Success Equation: Untangling Skill and Luck in Business, Sports and Investing. One of the many interesting nuggets in the book was that fund managers demonstrably exceed the number of multi-year "streaks" of outperformance we would expect to see if performance were truly random (identified in a paper by Mauboussin's son). 

Based on the data posted by Chase here, I built a shoddy, homemade version of a Monte Carlo simulator and ran 2000 versions of the actual teams/seasons. With a few small changes (the '99-present Browns are a sad, weak imitation of the pre-95 version and I won't have them associated with each other), I ran the simulations with the actual teams and years played. This ensured that we maintain equivalency (e.g., the 2003 Houston Texans can have a maximum streak of 2 because it's only their 2nd season and so on) and it was a fun chance for me to hit F9 2000 times because the iterations were not working.

The results are below but I found two parts very interesting. 

First, the outperformance was never significant at a 90% level outside of the first year, meaning there are fewer streaks started than would be expected. These extra seasons of outperformance show up in the form of slightly longer streaks (actuals average 3.96 years while simulated streaks average 3.76).

Second, there are some terrible teams in the NFL. The underperformance is significant at the 90% level from seasons 6 through 13 - far more teams are bad at drafting longer than we would expect by chance alone. For those interested, these teams are the Falcons (season 10 in 1990), Colts (1984), Cowboys (1987), Raiders (2007) and Chargers (2000). 

The next level analysis here is to segment by team management so that different eras are recognized as discrete units rather than part of a larger whole. We should really see some differentiation there as "good" GMs stay employed and rack up streaks.