The value of a Major League Baseball general manager (Review)

Lewis Pollis, a recently graduated senior at Brown University and past/future intern for several MLB teams, put together an analysis of the value of a general manager in Major League Baseball for his senior thesis in economics. He also had a summary version posted to Regressing, Deadspin’s analytics-oriented sub-site.

The paper estimates the value of the top-performing GMs to be tens of millions of dollars higher than that of middling GMs, who are likewise tens of millions of dollars more valuable than the worst performers. From there the paper links this value differential to the current narrow band of salaries and advances the notion that “the best baseball operations employees are paid substantially less than they are worth to their teams.”[1]

What is being measured in this research is how well teams perform. This performance is measured across a narrow set of activities (signing free agents, making trades) and attributed to the specific GMs who conducted the activity with some caveats about what can and can't be included as well as how it is measured. I am bought-in on this approach. These are hard things to measure and this seems to be a good way to get at it.

The outcome of this performance measurement is a distribution of value generated from best to worst and the conclusions noted above – the value that a good GM provides over a bad GM – flow from this distribution. This is where I have a couple of issues.

A couple of issues

For one thing, Pollis notes that the specific GM is not an accurate predictor of how a trade or signing will work out[2]. If a GM is not a predictor – or more accurately if making good picks in the past doesn’t make them any more likely in the future – then the value difference may be coming from luck rather than skill.

In the posts I did looking at the NFL draft I found no evidence for excess skill in selecting players. In the year-over-year data there were a few more streaks of bad selections than a purely random model would suggest, but the streaks of good picks were well within the expectations from the random model.

Another point offered in support of the model is that “subjectively speaking, the individual rankings seem more closely aligned with how well the GMs’ teams performed than with outsiders’ views of their decision-making processes.”[3] I would suggest that this is somewhat worrisome in that it implies the model may simply be a more complicated way of measuring success for teams – the model is drawn from wins above replacement, wins above replacement are highly correlated with team performance, team performance is what tends to drive GM reputations. Now I’m not sure there is a good way around this but I would be interested in any counter-intuitive results – maybe some GM has had bad injury luck in the draft but free agent signings and trades show a skilled evaluator.

Finally, the paper notes that the correlation between GMs’ measured ability in trades and measured ability in free agent signings is effectively 0[4]. If the model were measuring the combination of skill in identifying good players as well as skill in paying them salaries advantageous to the team – and this is what I understand the model is attempting to measure – we should expect those skills to be common across trades and free agent signings. I can’t think of a skill-based explanation for why a GM would excel at one and not the other. I can think of a non-skill-based explanation: luck.

Conclusions

I enjoyed this paper, and I think this is a great start at separating good from bad in GM performance. What I would like to see is more effort put into demonstrating that good or bad performance relates to the specific GM.This is far easier said than done and might require a lot more observations in the data set.

In the absence of additional data, the fact that skill across trades and free agent signings is not correlated leads me to suspect the answer is that GMs who measure highly were unusually lucky in these activities while those who measure poorly are unusually unlucky. This fits squarely into the paradox of skill – the importance of luck in determining outcomes rises as the overall group becomes more skilled. 

In baseball, as in football and investments, the decision-makers tend to be very smart people and the organizations have developed sophisticated infrastructure to evaluate talent. The result of this is that individual teams are not likely to be significantly more skillful in their evaluation than others and the outcome will increasingly defined by luck as the skill level rises. See here for a fuller explanation.

My biggest disappointment with all of this is that the next iteration is unlikely to be available for public consumption with Pollis now working in-house for the Cincinnati Reds. I hope he still manages to publish occasionally, and I wish him and the Reds good luck.

---

[1] Page 62

[2] Pages 47-48

[3] Page 49

[4] Page 49