Luck vs Skill - Is anyone good at picking football players?

A couple of weeks ago I took a look at the randomness in NFL Draft results, prompted by articles from Brian Burke and Chase Stuart on advancednflstats.com and footballperspective.com, respectively. I found some slim evidence for skill in drafting – the streaks of drafts with outperformance were slightly longer than expected. Interestingly there was much stronger evidence that some teams are not good at drafting, but you can read that post to get the details there.
 
The year-over-year data, however, has a major issue that I mentioned in the post: it’s not the same people making the decisions. Looking at the picks made by the Browns in 2012 and those made in 2013 provides no insight because the regime had turned over in the interim. Above and beyond that, the competitors do not behave similarly year to year so the decisions of each team – the variable we are trying to evaluate – are mixed in with changes in the way that other teams decide.
 
In response to these problems it seems like a study of the picks within a single year would be the place to look. The premise is that any team that has made a good pick should be more likely to make a good pick with their selection. If the initial pick reveals anything about their skill level, the subsequent success rate will exceed that of teams with a miss on their previous selection.
 
Using my data set of draft performance in the salary cap era[1] and my analysis of what a draft pick is really worth, we can quickly look at which picks were successful by identifying the ones that delivered more than the expected value. The percentage of picks that are successful varies widely by round due to scouting focus and the binary outcomes in later rounds[2]. If a player makes it and stays in the league for a couple seasons, they probably generate enough value to be a success because the expected return is so low in later rounds. Most of the picks in these rounds wash out: 269 of the 569 7th round picks play 1 season or less, and a grand total of 0 of them delivered value in excess of their expectation.


Streaking

The first way of analyzing picks will borrow from the methodology used in my first post on this subject (which incidentally borrowed from a study of mutual fund managers’ success highlighted by Michael Maubbousin in his book The Success Equation). Each pick has an expected success rate based on the average success rate in the sample – I use the success rate of a round which is somewhat arbitrary but reflects the way that team leadership could realistically be expected to approach the draft.
 
With this expected rate of success for each round I ran simulations of the real drafts 1,000 times so that the simulation set would have the same profile as the data set in terms of sequencing (e.g., the Browns drafted 6th, 70th and so on in 2013 and the simulation would have them do the same). To demonstrate, here is what 20 simulations of the Baltimore’s 2006 draft looks like with a 1 being a good pick and a 0 being a bad one.

Applied to the full data set, this methodology shows us how many streaks of a given length we should expect to see given the actual order of selection and average success rate for each round.

The results show no evidence for skill in drafting. The number of streaks we actually find in the data is well within the range of the number of streaks in the simulation for both streaks of good picks and streaks of bad picks.

One on one 

But wait, you say, isn’t it possible that since teams might spend a much higher amount of time evaluating early picks that all this streak stuff misses because the relevant streaks are only 2 picks long? Excellent question! To answer it let’s take a look at the individual picks by sequence and by round.  

There is not a lot to see here. The P-Values are all over the place and while a few are significant, looking at this many data points we are bound to find a few that are significant by chance. I have a hard time believing that the outcome of the prior pick is significantly correlated in the case of a team’s 8th pick even after being unrelated for the first 7.
 
On those extra picks in the first round we can certainly put together a story that includes a general manager freed of the pressure to conform that they felt with the first selection who boldly selects the underrated player they secretly knew all along would be a star, but it’s a very small sample size and the simulation standard deviations reveal them to be well within the range of the expected results given the average success rate.

A paradox

I don’t want to use this analysis to suggest that there is no skill in drafting – I don’t think 31 fans off the street would fare particularly well against 1 real GM with a full scouting organization – but there doesn’t appear to be excess skill. That 1 GM with a full scouting organization is competing against 31 other GMs with similar resources and precious few proprietary insights. All of them pore over the same game tape and combine stats save a few private workouts with some prospects.

To go back to Maubbousin’s book, what we have here is the paradox of skill in practice. As teams are very skillful at drafting there is less excess skill for any one team and the outcome is dominated by luck. Based on this data, there may be even less skill involved success or failure of NFL drafts than there is in success or failure of mutual funds.



Note - I went a step further and looked only at those picks made after a team traded up. See the analysis here.

[1] Looking at the drafts from 1994 through 2006 with a total of 3247 players [2] Hat tip to Football Perspective commenter Eric Barry for proposing this would be the case due to scouting focus on early picks