Some news is worse than no news: The NFL Scouting Combine and projecting draft picks

In general it is a pretty good assumption that adding information to a decision making process improves the outcome. In specific instances, however, the added information can be extremely disruptive. With respect to the NFL Scouting Combine, does it help? It depends…

The method

While we do not have visibility into teams’ draft boards before and after the combine, we do have access to one consistent record of pre- and post-combine sentiment from a consistent (and remarkably well-coiffed) source: Melvin A. Kiper, Jr.

By pulling several years of draft predictions (2004 to 2008) and evaluating the change in quality of predictions on either side of the combine, we can evaluate whether they improved. For each player, the absolute difference between the value they have delivered and the value of the player picked in that position will be the quality of the prediction: the lower the better. If, before the combine in 2004, Kiper projected Larry Fitzgerald to be the 1^st pick, then the quality of that pick would be the value provided by Philip Rivers as the actual most valuable draftee (64% as ranked by Career Salary Cap Value) less the value provided by Fitzgerald (52%). The average of all those individual values is the aggregate quality of the prediction[1]. A prediction with 0 error will have all players ranked in order of their actual career value to date.

Remarkably (or maybe not remarkably, he’s been doing this for a long time), Kiper shows an improvement at the aggregate level from before the combine to after. Across 152 individual players the average improvement in the prediction per player is 0.03% of the salary cap closer to the actual value of the players. Of course this 0.03% means that the average prediction improved by $38,000 of football value at the current cap number. The real action is looking at the distribution of increased (or decreased) accuracy for each position.

40 times matter, just not for everyone

Quality of prediction by position - 2004-2008

Change in quality of prediction by position - 2004-2008

Relative change in quality of prediction by position - 2004-2008

Defensive backs, wide receivers and tight ends see the biggest improvement in prediction accuracy from the information provided at the combine. Defensive tackles and offensive tackles, on the other hand, show a large degradation in the quality of predictions after the combine as compared to those before.

Running backs, interestingly, were way off before the combine and remain way off after the combine. There are too many cases like LenDale White, who moved up from 17^th to 10^th following the combine only to end up as the 122^nd most valuable pick of 2006, and Steven Jackson, who moved down from 14^th to 21^st and ended up coming in 12^th in value.

Drafter beware

Quality of prediction (or pick) by position - 2004-2008

Change in quality of prediciton (or pick) by position from after the combine to the draft - 2004-2008

The excitement doesn’t end there immediately following the combine. The average error from Kiper’s post-combine prediction is actually better than the average error from the draft itself (using draft position as “prediction” of value) for QBs, tackles and linebackers – all three show an increased error between the combine and the draft even after increasing following the combine. Teams would have more accurate picks at these positions going by the early-February predictions from Mel Kiper over their own draft board. Assuming that Kiper is relatively representative of teams' approaches - reasonable since his prestige depends on matching the actual outcomes - a better bet might be to just stick with tape of college performance and ignore the raft of new data that comes in the combine and pre-draft workouts.

On the other end of the spectrum, the tight ends, wide receivers and defensive backs continue to increase in accuracy between the combine and the draft. This is where teams appear to get a return on their time and energy getting players into the right slots. The new information drives significant improvement in accuracy of picks. The 4% improvement in accuracy (in terms of career salary cap value) would mean nearly $5 million in extra value for teams – not a bad improvement when the contracts for most first-rounders pay out between $5 million (32^nd pick, 2011) and $12 million (8^th pick, 2011) in guaranteed money. 

Running backs did show some improvement in this window. Chris Johnson, one of the all-time combine phenoms, failed to make Kiper’s pre- or post-combine prediction but was drafted 24^th overall and currently sits in 6^th place for the 2008 draft.

Finally, this raises a lot of questions about the accuracy of picking various positions in the draft. I will address these in a future post looking at both absolute error and the distribution (error can work out pretty well for a team sometimes).

Position counts within the 2004-2008 Mel Kiper sample, for reference:

DB          26

WR         22

TE           9

DE           20

G             5

RB           15

LB           15

QB          11

DT           14

T              14

C             1 (not shown in figures but counted in aggregate number)

---

[1] Using the values rather than simple ranks (Kiper predicted Fitzgerald to be 1, he was 5 so the error would be 4) automatically gives more weight to the top of the draft where elite talents should be more evident and mistakes are more costly. In the 2004 draft, the difference from 1^st to 10^th is 34% of the salary cap in terms of career performance to date while the difference from 20^th to 30^th is only 3%.