The value of free agents in the NFL - Part 2

The previous post in this series (check out Part 1 here if you missed it earlier) looked at whether teams do well based on how many homegrown players they have or start. There’s some small ability to explain performance but it doesn’t get to the level of detail needed to inform specific allocation decisions. Looking at how well individual players perform when changing teams, however, is a promising way to get enough information for just that kind of decision.

Player Performance

There are a couple trackable and quantifiable factors we might expect to predict a player’s performance in the next season. First and foremost is how they are performing this year. We will be using Approximate Value as created and calculated by the good folks at Pro-Football-Reference.com here as a proxy for performance.

As I qualify any analysis based on this data set, AV is necessarily and admittedly approximate (it’s in the name if you look closely). By using it in a data set as large as this we should avoid having random one-offs impact the results significantly. A great AV typically results from a great player on a good to great unit (offensive or defensive). A good AV means a player played a significant role on a good unit or a huge role on a middling one. A middling AV might be a decent player on a bad team or a rarely-used player on a good team. A low AV is a player who didn’t play or a player who played on a very bad unit.

Average AV by Performance
All-Pro	11.25
Pro Bowl (All)	9.88
PB (not All-Pro)	9.40
Starter (All)	6.87
Starter (no AP/PB)	6.34
Non-Starter	2.80

Next up is whether a player changed teams. It’s the point of the analysis and it would be a bit disappointing if we didn’t include it here. Those players who change teams should be expected to suffer a decline in performance.

We could conceivably see differences in the ability to integrate into a new team based on the position played with less complex positions potentially requiring less adjustment and yielding better performance for team-changers. To attempt to address this, we’ll split data set into four sets with one for each combination of offense/defense and skilled/unskilled and run the regression for each[1].

The age of a player and their number of years in the league might also be informative so we’ll add them to the regression[2]. They will interact somewhat but we’ll let the model determine which one reflects the positive bias in the data (see below) and which one reflects the inevitable decline of a player as they age. In place of a constant, these two terms will interact to create a base for the average player/years played combination. For example in this data set the average player in their first season is 23.33 years old, 24.32 in their second, 25.29 in their third and so on.



There will be a positive bias to this regression. Players who are not able to claim a spot on any of the 32 teams will simply disappear from the analysis rather than go to zero in projected AV. As the point of this is to examine the change in performance of players switching teams, a player that both the original team and all 31 others deem not good enough to play is outside of our scope. The lack of these players will probably push up the predictive power of previous season AV since the players who would decline most significantly – implied by the fact that no team picked them up – are not in the analysis.

Before digging into the analysis there are a few exclusions we need to make in the data set. Rookies, first of all, are out because they have no prior performance level and they have all changed teams. 

Of the remaining players we will assign them to one of two groups: prior starters or prior non-starters. Whether a player started the prior season or not impacts their prior AV (more play generally leads to higher AV) and provides significantly more information to the potential other teams. This gives us another dimension to split our data set (prior starter or not) in addition to the four already noted above (offense/defense and skill/non-skill) so we should probably move on before we split the data set even further.

One last thing before we actually take a look at the data. Even though we already excluded rookies the data are still influenced by players on their initial contract.

Figure 5 - Percent of Players Changing Teams and Year to Year Performance Change

There’s a lot to unpack in this figure[3], so focus on the left side first. In years 2 and 3 very few players change teams (the 5% and 9% columns) and those who do are way below those who remain on the same team in terms of performance relative to prior season (the lines, 48% vs -18% in year 2 and 8% vs -11% in year 3). You might say that this is what we expect, given the hypothesis that signing free agents is a less effective way to compete. Unfortunately there are structural complications that make these seasons bad comparisons. The CBA in effect over this period prohibited players with fewer than 3 accrued seasons from negotiating with other clubs as long as their existing team made them a required one year tender at the minimum salary[4].

Players in year 4 are also problematic due to restricted free agency, in which the player could negotiate with any team but the existing team retained a right of first refusal and, potentially, draft pick compensation from the team that signs the player. Restricted free agency was also trending down throughout the period up to 2009, which was the first season since its inception in 1993 that no players changed teams under restricted free agency. Just to be safe we’re going to also exclude all of these players from the analysis of performance after changing teams. With the new CBA putting in place 4 year contracts for all drafted rookies, the pool of players in the future with the potential to change after 3 years in the league will be limited to undrafted players only.

Those 11+ season players off to the right of the chart who have a jump in performance when they change teams are a relatively small part of the data set. They represent just 5% of the seasons and about 10% of the team changes. Someday we'll take a deeper look at them, just not today.

The Data

Now that the qualifications are out of the way it’s time to look at the data. We’ll start with the regression for all Prior Starters just to make sure everything’s working.

Figure 6 - AV Regression for Prior Starters

This not only appears to be working, it is working pretty well. The regression for Prior Starters has an R^2 of 0.28 and all of the variables included are significant. As expected, the Age and CareerSeason variables cancel each other out to a net of -0.09 AV per additional season. One way to think about this is to interpret the constant plus the Age and CareerSeason variables as a baseline for the average player. For the prior starters, we would expect the following assuming they entered the league at 23:

Figure 7 - "Baseline" AV Performance (Prior Starters)

The projected difference from this line – which again assumes that the player was a starter the prior year and is still in the league in the current year – would be the addition of 54% of their AV from the prior season. If the player is still on the same team we stop here. If, however, the player has changed teams, we take off 16% of the player’s AV from the prior season so the net is the baseline plus 38%. Changing teams takes away 30% of a player’s difference from the baseline.

Figure 8 - Prior Starters (Changed Teams)

Figure 9 - Prior Starters (No Change)

We can see here visually what the regression is telling us. The players who change teams are much more likely to end up with a 0 or 1 AV than the players who don’t. Starting from a lower average (5.76 vs 7.40 for those who don’t change), the team changers end up dropping 1.23 units of AV compared to the 1.00 units lost by those who stay on the same team[5].

The right side of the curve appears to offer some hope for the team changers. The proportion of elite players in Year N+1 seems about the same as in Year N. If we look at those with AV 10 or more in the prior season, those who change teams have an average AV of 12.22 against 12.90 for those who don’t change. The team changers are about a year older (30.7 yrs old vs 29.8) and there are only 41 of them in the sample compared to 542 who don’t change. The AV of those team changers declines by 4.32 the following season while those who stay on the same team decline by “only” 3.35. Looking back at the Age/CareerSeason baseline from earlier it is clear that the additional year does not account for the difference.
 

Prior Starters	Coefficient				P-Value
	Skilled		Line		Skilled		Line
	Offense	Defense	Offense	Defense	Offense	Defense	Offense	Defense
Intercept	8.23	13.37	8.76	11.09	0.00	0.00	0.00	0.00
Prior AV (All)	0.65	0.40	0.52	0.43	0.00	0.00	0.00	0.00
Prior AV (Changed)	-0.19	-0.09	-0.23	-0.15	0.00	0.06	0.00	0.00
Age	-0.28	-0.45	-0.25	-0.31	0.00	0.00	0.01	0.00
Career Season	0.21	0.38	0.18	0.19	0.03	0.01	0.09	0.08

Prior Non-Starters	Coefficient				P-Value
	Skilled		Line		Skilled		Line
	Offense	Defense	Offense	Defense	Offense	Defense	Offense	Defense
Intercept	4.51	-1.34	6.49	8.44	0.00	0.66	0.01	0.00
Prior AV (All)	0.55	0.23	0.28	0.68	0.00	0.10	0.13	0.00
Prior AV (Changed)	-0.14	-0.14	-0.15	-0.16	0.04	0.26	0.45	0.16
Age	-0.12	0.16	-0.14	-0.28	0.04	0.23	0.20	0.01
Career Season	0.05	-0.15	-0.04	0.23	0.48	0.28	0.76	0.04

 
 Note: Those values that are bold/italicized are where p>0.05 for that coefficient

	Prior Starters				Prior Non-Starters
	Skilled		Line		Skilled		Line
	Offense	Defense	Offense	Defense	Offense	Defense	Offense	Defense
R^2	0.40	0.16	0.25	0.18	0.19	0.02	0.06	0.10

Just as with the full Prior Starters group the combinations of Offense/Defense and Skill/Line all show substantial decreases in projected performance for players who change teams.

The proportion of the decreases is much higher on offense, possibly as a consequence of the more-complex playbooks on that side of the ball. Without trivializing what goes into a defense, it passes the smell test that a cornerback can be incorporated into a new scheme more easily than a wide receiver. After learning their role in blitzes and other specific calls the cornerback might not have to adapt at all in the case of a man coverage scheme or might have to adapt to the styles of the other defensive backs to fill a role in a zone defense. The wide receiver needs to learn all of the plays in the offense and, at the same time, get up to speed on the specific way their new quarterback reads the defense to ensure that they are on the same page for how to adapt to the defensive calls.

At the same time the defensive players exhibit a much greater regression even without changing teams. The coefficient of Prior AV for a skill position player on defense (0.40) is less than that of an offensive skill player who changes teams (0.65 – 0.19). Because AV rewards individual statistics (sacks, interceptions, TDs etc.) the metric is much less persistent for defenders because defensive statistics (sacks, interceptions, fumbles and the All-Pro and Pro Bowl teams that follow those metrics) are much less persistent than offensive statistics. It's also very possible that defensive players are more prone to injury than offensive players. These factors are a limitation of AV and a reflection of the luck embedded in football. As a decision maker, the fact that these stats are less persistent should cause you to discount the gaudy sack or interception totals of a player somewhat since these are a combination of the player’s skill and the luck they experienced.

So it seems pretty clear that players who change teams perform worse than they otherwise would have based on their age, experience and prior performance. That’s one step closer to proving that teams should not be investing in them.

The next piece of the puzzle is incorporation of value. It may not matter if these players decline in performance if teams get a good deal, or even a bad deal that’s better than the ones for players who re-up on the same team. 

Check out Part 3 here to look at value

Check out Part 1 if you missed it earlier



For those of you who can't get enough of the data - the output from each of the regressions (Starter/Non-Starter, Offense/Defense and Skill/Line combinations) is below. Enjoy.

 




 

 

---

[1] The following positions will be considered “skilled” for this analysis: QB, RB, TE, WR, DB. Fullbacks are mixed in with running backs (all categorized as RBs) in the data set so they will be considered “skilled.”

[2] I also considered including a variable for tenure with a team, thinking that longer tenure would lead to better performance. This variable ended up interfering with the “Changed Teams” variable for obvious reasons so it is out. It is also difficult because teams tend to keep good players so a variable for tenure would have a big chance at a positive bias above and beyond the one already noted in the analysis.

[3] Because I made it needlessly complex by combining two graphs into one.

[4] The definition of "Accrued Seasons" comes from the 2006 CBA: “For the purposes of calculating Accrued Seasons under this Agreement, a player shall receive one Accrued Season for each season during which he was on, or should have been on, full pay status for a total of six or more regular season games”

[5] It’s not surprising that both groups saw a drop because one of the components of being a starter is being reasonably healthy. The Prior Starters group is disproportionately filled with those who had good injury luck the previous year. When they have average injury luck the following year we see regression.