The Worth of a Rivals.com Star: Stars Matter in College Football

There are numerous debates on the Rivals network regarding whether there are differences in how good a particular recruit is.  For example, team X gets a verbal commitment from player A who is listed as a 4-star player, while team Y gets a recruit from player B who is listed as a 3-star player.  Internet bickering/flaming ensues about who got the better player.   Further, this debate sometimes continues along the lines of “Well not every player pans out . . .” and “Lots of 5-stars never make it (whatever that means) so that being a 5-star is not a guarantee of success”.  Similarly there are lots of successful 3-star players.  There’s anecdotal evidence on both sides.  So we thought we’d look at some data, since that’s what this blog was started to do: Answer sports questions with real data.

Data
From the Rivals.com list of commitments for all current ACC, Big Ten, Big XII, Pac-10 and SEC members from the recruiting classes of 2002 and 2003.  Transfers were not included.  Current Big East schools were excluded because some of the current members were not members of a BCS conference in 2002 or 2003.  That data can be accessed from here. For each player we obtained their position, number of stars (based on Rivals rankings), school to which they committed and the year they committed.  We then needed a measure of success against which to compare how these players performed (i.e. did they make it.)  As our metric we chose whether or not a players made their all-conference team (First or Second team) for any of the years 2002 to 2007.  Where possible, we used the coaches selections but in some cases it was necessary to use the AP all-conference squad.  (It was striking how hard it actually was to find some of these all-conference selections even with both Lexis/Nexis and Google searches.)  This is not an ideal metric since players can be very valuable members of their teams and not achieve all-conference status.  Collecting data such as the number of starts a player made would be a viable though much more time consuming endeavor and would probably result in similar outcomes since there is little doubt that these measures would be correlated. 

There were (n=)2580 total players that were considered. Of these 2580, 330 were selected all-conference for a success rate of 330/2580 = 0.1279.  Below is a table of the data broken down by position.  It is somewhat surprising that so few RB’s were selected though it is possible that many of the players listed as ATH (athlete) became running backs.  Additionally, this is also probably reflective of the movement to single back offenses.  Their number of DB’s is perhaps reflective of the need to get quality DB’s in a more passing oriented game.

The breakdown for stars is given below:

Not too many surprises here, though it is interesting to note that there were actually 19 players taken by these major conference teams with one or zero stars.  One might suspect that these were punters or kickers but none of the 19 are listed as kickers. 

It might seem surprising that the Big Ten and Pac-10 seemingly have a good bit fewer players taken than the other conferences; however, if one looks at the number of players per conference member then it is clear that the number of players taken per team is between about 42 and 48 for these two years.  The Big XII (48.2 per member)  and the SEC (47.4 per member) are higher than the ACC (42.2 per member) and the Big Ten (42.7 per member) with the Pac-10 (45.7 per member) falling in between. 

Results
Our primary interest is in the relationship between stars and making all conference (AC).  This result is given below

We note that the differences in probability of AC between those with 5-stars, 4-stars and 3-stars are all statistically significant (p<0.01).  We didn’t look at whether or not differences between 0, 1 or 2 stars exist.  We then looked at whether there were differences between the probability of AC for players from different conferences and for different positions.  Using a logistic regression analysis, once stars were included in the model, the other terms were not significant with one exception:kickers.  Kickers tend to be all-conference at a higher rate than other positions with the same number of stars.  

Conclusions
STARS MATTER.  It is pretty clear from this short study that the number of players that Rivals assigns to a player do predict performance (at least as measured by AC).  Further there does not seem to be a difference between the conferences, nor between the positions on the likelihood of being an all-conference selection.  The exception to this is kickers.  It is likely that there is a potential bias involved in this assessment since it is probably only the best kickers each year that are rated and offered scholarships.  Only four kickers were given a 4-star rating and one were given a 5-star rating.   Perhaps a better way to think of it is that Rivals underrates kickers.  It’s unlikely that Rivals is alone in this.

A 33% or 34% success rate might be good in baseball (okay in baseball its HOF) but it is good enough here?  That’s a hard one to know.  Clearly we are evaluating and projecting future performance of 17-, 18- and 19-
year olds.  No doubt that is hard.  Lots of variability here is due to maturity (both physical and mental), work ethic, perhaps even intelligence.   So here’s a thought.  Is Rivals (or Scout or ESPN) getting better at rating recruits?  That would be a idea worth tracking.  It would also be looking at the individual rankings (#8LB, #12QB or #56DB at a position to see if that is predictive of performance.

Ideally, it would be best to find better metrics for player performance though no single measure will be ideal.  We limited our study to two years worth of recruiting classes (2002 and 2003) and a single recruiting services ratings (Rivals.com).  If those years were anomalous, then it is possible that ratings do not matter.  We think this is unlikely.  There are alternatives to Rivals.com and their rankings are different each year.  We would expect that another set of rankings would yield similar results.  What would be more interesting is to look at a larger database of information on each recruit and to see if there are traits (40-time, high school gpa) that can better predict who succeeds and who doesn’t.  Further, coaches ought to be thinking about whether or not their recruiting is improving (more efficient, more productive) over time.  Its an insanely competitive environment and even modest improvements should pay dividends.  Such a study would be a major undertaking but it is just the type of work that Statistical Sports Consulting does on a regular basis.