So the topic of Quality of Competition (QoC) came up at the MIT Sloan Sports Analytics Conference recently and it has come up on twitter a couple of times since as well. Along with Quality of Teammates, there have been discussions along these lines since I joined the hockey analytics community via the yahoo group HAG (hockey analytics group) quite a while ago. And the issue, in my opinion, is framed the wrong way from a statistical viewpoint. Below I’ll discuss my, admittedly mathematical, approach to this.

The traditional way that hockey analytics refers to QoC and its partner Quality of Teammantes (QoT) is as single quantities. This suggests that what is being referred to ** is **a single quantity. Generally across multiple games, it’s not a single quantity but it gets talked about that way. This approach results in unironic statements like ‘quality of competition doesn’t matter’. When you apply that to, say, facing a Capitals line of Ovechkin, Oshie, Backstrom, Alzner, Carlson versus the 3

The notion of the value of Quality of Competition (QoC) or more explicitly average aQoC is a crude one. I’m not fond of QoC because the way that we should be dealing with assessing players is by estimating their impact on all of the events we are interested in **simultaneously**. The use of QoC comes from trying to analyze data at the player level rather than at the event level. The scale matters for data analysis. One tenet of Statistics, the discipline, is that your unit of analysis should be the largest unit that is impacted by your factors. In the case of sports that often means plays or shifts or possessions. Consequently, doing analysis at a game or season level is not ideal since it doesn’t disentangle all of the important impacts.

Since players skate against multiple opponents over the course of games, there is a need to address the impact of opponents on a player and the outcomes that results. One way to do this is to average those impacts, which is what most QoC do. And they do that by taking raw counts and averaging. The primary issue is that if you have players who are impacted by aQoC or aQoT effects then there raw numbers will have that impact and it will permeate your estimates unless you adjust for them. For example, Connor Sheary, when healthy, has been playing quite a bit with Sydney Crosby. If we are interested in Sheary’s aQoT for xG, then a calculation of that aQoT by averaging the individual player xG’s for his teammates weighting for event together will include Crosby’s xG impact. Now Crosby’s xG impact will include Sheary’s xG impact as well. If they are both players that impact xG positively then the aQoT xG for both players will be higher than reality since we have not extracted their individual impacts. That is, we need to have a way that extracts Crosby’s impact on xG (or any other outcome you want) while adjusting for Sheary and Hornqvist and Dumoulin and anyone

Could WOWY’s save us? Maybe, but then we will create a string of comparisons (Crosby without Sheary but with Kunitz, Crosby with Kunitz but without Sheary, Crosby with Sheary but without Cullen, Kunitz with Crosby but not Cullen, etc.) that is very hard to disentangle and adjust. And the thing about WOWY’s is that they assume everything else is equal when you make the with/without comparisons. Fortunately there is a better way.

It is possible to address QoC and QoT systematically at an appropriate level for analysis. One way to do this is through a multiple regression that has a term or terms for each players on the ice for a given event or shift. This area of research is known as adjusted plus-minus in the statistical analysis of sports literature. It is not easy to construct and can take some compute processing but it is the right methodology for this type of analysis. Begun by Rosenbaum (2004) (http://www.82games.com/comm30.htm) for basketball, examples using this framework has been published for hockey by Macdonald (2011, 2012), Gramacy et al. (2013), Schuckers and Curro (2013), and Thomas et al (2013). There is no need for an aQoC ora QoT to help with these approaches since these models explicitly assess and adjust for the impact of each player on each event or on each shift.

Here’s how this works for a game. Let’s consider how you might use regression to adjust for other players on the ice by looking at the play by play for Game 1 of the Stanley Cup Finals. Here is a link to that file: Game 1 2016 Stanley Cup Finals Play by Play file. See the screenshot below and note that the events are numbered in the first column on the left.

The first action event (i.e. not a period start) is #2, a faceoff at neutral ice with SJ: Thornton, Pavelski, Hertl, Martin, Burns and Jones and PIT: Crosby, Hornqvist, Kunitz, Dumoulin, Letang and Murray. This is won by SJ and so value, if any for that event would be given to the SJ players and taken from the PIT players. Note that we record each player that is on the ice for that event.

Before the next faceoff (and after a stoppage which we will ignore), SJ makes a change and it is now Marleau, Couture, Donskoi, Vlasic, Braun and Jones, while PIT still has the list above. We would record the change of players in our data and credit/debit the appropriate players for being on the ice for the subsequent events, here a faceoff win by SJ and a hit by SJ.

The documenting of players on the ice usually takes the form of a row in a large matrix (called the design matrix, X) where each column is for a player who has appeared in a game and each row is an event or shift. If the player is on the ice they receive a 1 if playing for the home team and a -1 if playing for the away team. Thus, in the above sequence we would have 1’s for the SJ players and -1’s for the PIT players. For any player not on the ice, e.g. Malkin or Wingels, there would be a zero in their column for the rows of the events above. Malkin and Wingels will have non-zero values in their columns for later shifts in this game. If we were analyzing all of the playoffs in this manner we would have columns for players for DAL and TBL and they would have zero’s for this entire game. Thus at even strength we end up with 12 non-zero entries per row of our design matrix.

Back in Game 1, the Marleau line is out for three events (# 4 through 6) but PIT makes a change after the 2nd of these, though Crosby stays on the ice for the 3^{rd} of these, #6. All of these plays are recorded as rows with players on the ice denoted by non-zero values in the appropriate columns. We continue the above for all of the relevant events in this game. Then we repeat the process for every game. For the analysis of an entire NHL regular season, there are about 1100 players and 250000 plays at even strength. Thus our design matrix will have 250000 rows and about 1100 columns. If we include special teams, there are another 50000 events. This is big data hockey analysis. By equating outcomes with impacts of players and other factors we have something like (n=) 250000 equations and (p=)1100 unknowns and we can solve that. Just a bit more complicated than your high school algebra of three equations and three unknowns, but a similar process for solving. Our solution, which are estimated player (and other factor) impacts, generally is one that finds a solution that minimizes the errors in those 250000 equations.

The assigning of values to the outcomes of on-ice events has been done a couple of ways. Macdonald (2011, 2012) — these analyses are now five plus years old — uses shifts and assigns values to shifts. That is, each row is a shift and the value of the row is the total value of that shift. Schuckers and Curro (2013) assigns value by play. Thomas, Ventura, Jensen and Ma (2013) looks at the rates of goal scoring by shift combinations and so as with Macdonald’s work the rows represent shifts or shift combinations of players. Gramacy et al (2013) use goals as success and saved shots. Not all of these methods use every event in the play-by-play files. For example, Macdonald does not use faceoffs in his models while Gramacy et al uses only shots.

All of the above methods start with work on even strength since that is the easiest to process but can be extended to account for special teams situations. In addition, we can have other terms in the model for score differential or for where a shift starts (zone starts) to account for those in our assessment of player impact.

Having defined the design matrix to have players from both teams, the resulting estimates of player impact via regression of the outcome metric with the columns of the design matrix accounted for their teammates and their opponents. The impacts of other players be they teammates or opponents are already in the estimates because they are already in the equations we are solving. The estimated player impact that we get is the average impact of that player on all the outcomes for which they were on the ice accounting for the players with them on the ice for all of those same events. That is, the methods deal with teammates and opponents on every event or every shift and, consequently, there is not a need to deal with aQoC or aQoT (or Zone Starts or Score Effects, etc) since they are built into the process. That said, it is possible to get values for average QoC and average QoT from the adjusted player estimates and the design matrix.

(Note none of the above regression adjusted plus minus methods for hockey use an ordinary least squares regression approach but they fundamental approach is similar.)

**Optional Math Content**

So it is possible to get an average QoC or QoT using the regression adjusted plus-minus framework that is part of the papers listed above. If we want to calculate QoC or QoT for these regression type models, we can. To do that define the design matrix as X, the outcome vector as **Y** and the estimated effects vector as **q**, where **Y** has length n and **q** has length p. Then we get estimated effects from

** q** = (X’X)^{-1}X’**Y**. (1)

The X’X matrix is a p by p matrix (and I’m using ‘ (prime) to represent transpose of a matrix). Focusing on the terms for players only, each element contains the number of shifts or events when players are playing together. For event data, this is straightforward. (For responses that are rates there is a weighting that might need to be done but that can be done accommodated pretty easily with a weight matrix.)

To get aQoC let A be a truncated version of X’X where all of the positive elements of X’X are turned into zeroes. Then let **a_ _{k}** be the kth row of A. To get an average QoC for the player represented by the kth row of X’X, we would take the

To get aQoT let B be a truncated version of X’X where all of the negative elements of X’X are turned into zeroes. Then let **b_ _{j}** be the jth row of B. To get an average QoT for the player represented by the jth row of X’X, we would take the

For the above, I’ve assumed that there are no other factors in X other than players. To get aQoC if there are other factors, the elements of a_k should be zeroed out for those entries. While I’m at it, we should be doing some sort of shrinkage rather than the ordinary least squares (OLS) in Equation (1). All of the aforementioned papers do this to improve their estimates of the impacts of these various factors, **q**.

This year registration will have a fee to cover lunch.

Full conference details are here:

www.statsportsconsulting.com/otthac17

Registration can be found here (note that registration is two steps: registration and payment)

http://forms.carleton.ca/math/2017-ottawa-hockey-analytics-conference/

]]>Below is a paper that we (Timo Seppa, Michael Schuckers and Mike Rovito) recently wrote on NHL Draft Analytics entitled **Text Mining of Scouting Reports as a Novel Data Source for Improving NHL Draft Analytics. **

Here’s a link to the paper: TextMiningScoutingNHLDraftAnalyticsFeb2017

Timo will be presenting this work at 2017 Ottawa Hockey Analytics Conference #OTTHAC17 on May 6th.

]]>Info:

Based upon a smoothed monotonic regression (monreg package in R) using the first seven years (post draft) of time on ice (TOI) for players drafted at each pick in the NHL draft (through the 210th pick). The data used for this draft value pick chart (DPVC) are the 2003 to 2008 NHL Draft classes. The chart is scaled (and rounded) so that the first pick has value 1000. A graph of the chart is given below.

I’ve previously done two charts for assigning value to draft picks. The first and still most popular was done in 2011 and used games played as the outcome measure for players drafted from 1988 to 1997. That is here: 2011 DPVC (As always in these sorts of analyses we have to use historical data.) More recently in 2014 I put together a DPVC for an early version of a paper I published with Steve Argeris. While the focus of that paper was evaluating how teams draft relative to the NHL’s Central Scouting Service, we put together a DPVC based upon the first seven years of TOI for players drafted from 1998 to 2002 based upon the idea that teams generally have rights to a player for seven years after they are drafted. The DPVC did not make the final version of the paper which appeared in the Journal of Sports Analytics in 2015. Here’s the link to the original version of that paper with the DPVC: Schuckers and Argeris.

The 2016 version above uses the same methodology as the Schuckers and Argeris paper but using first seven years of TOI from the 2003 to 2008 NHL draft classes. The basic idea of a DPVC is to develop a value on a particular draft pick to have a starting point for possible trades. We have used a statistical method that forces the value of draft picks to be decreasing because that is what I believe inherently should be the structure of a DPVC. (Even though, for example, in these data the average TOI for the 9th pick is higher than the average of the 8th pick.) I just don’t think that you should value a pick which give you fewer choices to have higher value than one that gives you more choices. As I said whenever I’ve talked about this chart, it is a guide and you have to also consider other information you have about the draft and the players available.

David Wilson, a student at Carleton University, who I co-advised has noted that there is some evidence that players selected in the fifth round outperform those taken in the 4th round, at least for the drafts from 1998 to 2008. David presented some of this work at the Vancouver Hockey Analytics Conference. David is the second speaker in Session 5.

**Methodology Details:** I’ve used a small bandwidth both for density and kernel estimation in order to capture edge effects, 0.04. I went to small values of these parameters while also trying to maintain smoothness. This was also done to try to avoid edge effects on lower, i.e. earlier, draft picks. For fitting of these data, I used all of the rounds for 2003 to 2008 to avoid some edge effects at higher draft picks. That is, I used, for example, all 292 draft picks from the 2003 draft and all 230 picks from the 2005 draft. Here is the command I’ve used in R: `monreg(DraftPick,First7TOI,t=210,hr=0.04,hd=0.04,degree=1)`

**POSTSCRIPT (June 8, 2016): Note about differences from the 2011 DPVC to the 2016 DPVC**

I’ve gotten some questions about the above chart and in particular the differences between this chart and the 2011 version (here). The 2016 version is steeper than the 2011 version. There are several differences that could account for this. First, in the 2011 version I used career games played (GP) as the response while in the 2016 version I’ve used TOI in the first 7 seasons. So it is possible that the different responses would result in different weightings for draft picks and different curves. Second, while obvious we are using different data for the two time periods — players taken in the 1998 to 2002 and the 2003 to 2008 drafts, respectively — so it is possible that there has been some changes in how players’ careers have gone during these time frames. Third,

Figure 2 below addresses the two issues from the paragraph above. For this graph, I’ve used the methodology described in the post above to fit the first seven GP and TOI data from ’98 to ’02 and from ’03 to ’08. The TOI curves are virtually identical. The blue is almost completely obscured by the red. (Note that both are scaled to have 1000 as the maximum values.) Similarly the differences between the the GP curves are very similar. There seems to be some slight differences in the first 20 picks and in the last 30 or so picks but given the likely variation in the curves they seem quite similar. Further, while there are differences between the GP and the TOI curves, their basic shape is pretty close. (This is not a surprise if we note that the correlation between first seven GP and first seven TOI is 0.932.) So the differences in the curves is not due to the different time frames and not due to the different response variables.

The difference seems to be due to the differences in the methodology that I used to create the two curves. Figure 3 below illustrates this issue which is somewhat a function of restricting myself to a monotonic (continuously decreasing) curve and somewhat a function of the smoothing parameters. The response data in Figure 3 is the TOI data from 03–08. The blue curve is the version given in the 2016 DPVC, the other versions are from taking different values of the smoothing parameters within the monreg function. The lightest grey one, topmost curve over the first 50 picks, is the one that seems closest to the 2011 version. (Technical note: In 2011 I used the loess function to make the DPVC while I am now using monreg.) This time I’ve gone with a curve that is a bit less smooth but, hopefully, better reflects the data. Ultimately I am making a choice here between overall smoothness and fidelity to the data. Inherently I believe that a DPVC should be relatively smooth and so I’m not inclined to have something with steps like the black line of Figure 3 (even less smooth than the blue line) but I want to keep with the data so I’ve moved from the light grey line which is very smooth.

]]>

The conference will be held January 16th, 2016 at Porter Hall/231 University Centre Carleton University in Ottawa, Ontario. The conference webpage (including registration) can be found here:

http://statsportsconsulting.com/ottanalytics16/

Michael Schuckers, professor of statistics at St. Lawrence, who will speak at the conference, says hockey analytics has become a way to better understand the factors that impact the outcome of hockey games and hockey seasons. “The conference is an opportunity to meet and hear from some of the top minds in hockey analytics,” he said. “This year, we are especially excited to have IBM Analytics as a sponsor and presenter and to have a session on player tracking, which will cover work on capturing player location data during games. For the casual fan, this is an opportunity to learn more about the sorts of analytics that are being used to evaluate teams and players.”

One of the highlights of the conference will be a session on player tracking data and its role in the future of analytics.

This year the lead sponsors for the conference will be Carleton University and the Ottawa Senators. The other sponsors for the conference are IBM Analytics, St. Lawrence University (Canton, NY) , American Statistical Association Section on Statistics in Sports, Carleton University Athletics and the Statistical Society of Ottawa.

There will be a social event for conference attendees at Sens House the night before the conference.

This 2nd annual Ottawa Hockey Analytics Conference will consist of talks on the analysis of hockey along with time for socializing and networking. The event will be similar to hockey analytics events previously held in Calgary and Pittsburgh.

The webpage for last year’s version of this conference can be found at: Ottawa Hockey Analytics Conference 2015.

]]>So there has been a bunch of discussion lately about tanking in the NHL. Seems pretty clear that this has happened before and is happening again this year. Puck Daddy doesn’t mind it. Grantland’s Sean McIndoe calls it a ‘disastrous situation’ and has some suggestions including one (Option 11) for a postseason tournament. The idea of a postseason tournament is nice in theory but I wanted to sit down and see if it was possible to build a tournament which gave worse teams better odds of winning the tournament. So I set out to put together a concrete proposal for a tournament. Though it would clearly need NHLPA approval, that tournament could run concurrently with the Stanley Cup playoffs. It might even be good to have it start after a small layoff when there are fewer games per night. Sponsorship might be fun: The Second Cup, anyone? Or the Last Chance Cup sponsored by the New York Lottery.

Below is the full proposal for a single elimination tournament to determine the first round NHL entry draft order. The idea here is to award the first pick in the NHL draft to the team that wins a tournament. I’ve laid out the full proposal in the pdf below. The key to the process is to design a tournament that will have the highest probability of winning going to the worst team. That takes some rethinking of the usual tournament design. This proposed tournament is represented in the graph at right where the worst team is listed as 14, the second worst is 13 and the best team (to not make the playoffs) is 1.

- Draft order is determined by how far a team progresses in the tournament.
- There are guarantees that the worst team will get no worse than the 3rd pick, the 2nd worst team no worse than the 4th pick.
- Games are played at the rink of the lower seed, again to provide worst teams with better odds of progressing through the tournament.
- Draft ordering could be different for rounds beyond the first.
- It is possible to extend the model to tournaments for 15 or 16 teams when/if expansion occurs.
- There would be additional revenue from ticket sales and TV rights.
- Playoff OT rules apply though I would be open to the possibility of using the shootout to determine a winner if it would benefit worse teams and I suspect that it does (unless you’re NJD circa 2014).
- Players eligibility for the tournament is the same as for the Stanley Cup playoffs.

As McIndoe noted in his Grantland piece there are some potential issues including questionable integrity of games. One remedy might be to consider monetary incentives — new Honda’s for every member of the winning team, maybe — created for players by rewarding players and teams financially for continuing to win. There are some tiers here for teams and so it is possible that a team in the 8th worse spot might ‘tank’ for the 9th spot.

Here is a pdf of the proposal with some additional verbage.

NHLDraftSingleEliminationTournament

]]>

Last night I did a webinar on Statistical Methods for the Analysis of Hockey for St. Lawrence University Alumni and friends. I had a blast. There were lots of good questions and lots of interest. Exactly what I would expect from a bunch of Laurentians.

Below is a link to the slides for the talk.

]]>Additional Details can be found on the workshop website below:

Workshop website: HERE

]]>

Our aim is to build a model which accounts for the relative differences between rinks on the events that are recorded. The rest of this post is a summarized version of our paper on this topic which is linked below. **The focus of this work is in making data recorded for the following events comparable from rink to rink: Blocks, Giveaways, Hits, Missed Shots, Shots, and Takeaways.** We also look at the recording of aggregated events that count as Corsi events, Fenwick events and Turnovers. The last of these was created by Schuckers and Curro to account for the home bias of Takeaways and Giveaways as part of the THoR paper.

The data that we use for this analysis comes from the nhlscrapr R package created by Thomas and Ventura and includes 6858 games from six regular seasons.

We used a statistical regression to model counts of events per game with several predictors **including team factors, average score differential, and rink.** The estimated rink effects that we derive can be used to reweight recorded events so that can have comparable counts of events across rinks. Applying our methodology to data from six regular seasons (2007-08 through 2012-13), we find that for the most part NHL rinks (and the individuals therein) do a reasonably consistent job of recording events. This is especially true of the recording of SHOTs, which has the fewest rinks with significant recording issues and has the smallest rink effects that we found. We only found that St. Louis and Florida have rates for the recording of rinks that differ significantly from other rinks. Florida counts shots at a rate that is about 3% higher than other rinks while St. Louis counts shots at a rate that is 4.5% lower than the rest of the league.

However, there are some rinks with rink effects that are significant and consistent across these seasons for other events. Zeroing in on blocks, hits and misses, there were four rinks that consistently inflated or deflated the counts of those events: Edmonton, Los Angeles, New Jersey and Toronto. There were also six rinks that were not significantly different from the rest of the league on those events. These were: Buffalo, Nashville, Pittsburgh, St. Louis, Tampa Bay and Vancouver.

While event counts are impacted by rink effect, ratios of events such as Corsi For Percentage remain relatively unaffected. The table below demonstrates how small the impact of rink effects are on the Corsi For %. This despite our estimation that Boston, Columbus, Edmonton, Los Angeles, New Jersey, and Toronto all have significant rink effects for the counting of Corsi events. New Jersey is particularly egregious, undercounting Corsi events by about 16%. The reason that ratios are not impacted as much as counts is that the impact of a given rink is felt in both the numerator and the denominator for the home team.

**Table 1: Comparison of Corsi For % and Adjusted Corsi For % for 2012-13 NHL Regular Season**

Top 5 and Bottom 5 teams on Corsi For Pct.

Team | Corsi For Pct. | Adjusted Corsi For Pct. |

L.A. | 0.5630 | 0.5628 |

N.J. | 0.5592 | 0.5592 |

BOS | 0.5433 | 0.5430 |

CHI | 0.5414 | 0.5420 |

DET | 0.5366 | 0.5365 |

… | … | … |

CBJ | 0.4711 | 0.4706 |

NSH | 0.4668 | 0.4662 |

BUF | 0.4512 | 0.4513 |

EDM | 0.4458 | 0.4445 |

TOR | 0.4408 | 0.4398 |

While ratios are not affected in a major way by rink effects, the counts of individual events recorded as part of the NHL’s RTSS system are. To illustrate this we looked at the Block shots recorded by players during the 2012-13 NHL Regular season. For those rinks where there are rink effects for block shots, we take each block and weight it by 1 divided by the rink effect. So that for Nassau Coliseum home of the New York Islanders, NYI, each block counts as 1/1.208 or 0.828 of a block since that rink overcounts blocks by about 20.8%. Similarly, since the rink effect for Anaheim is 0.721, each block in that rink counts as 1/0.721 or 1.387 of a block.

We adjusted every block during the 2012-23 regular season for the rink in which it occurred and the top 10 players on adjusted blocks is given in the table below. Unlike the case for Corsi For %, there are substantial changes for the counts of block events. Here the top player on the revised list, Francois Beauchemin, was originally ranked 10th on the list of players with the most blocks. There are other large changes for players from the Islanders (Andrew MacDonald) and the Capitals (John Carlson) due to the relative counting of BLOCKs in those rinks. Macdonald and Carlson move from tied for third in raw BLOCKs to eighth and ninth, respectively, after our adjustment.

**Table 2: Comparison of Raw Block Counts and Adjusted Block Counts for 2012-13 NHL Regular Season**

Top 10 players based on Adjusted Block Counts

Player | Team | Adjusted Blocks | Raw Blocks | Differential |

F. Beauchemin | ANA | 133.3 | 111 | 22.3 |

G. Zanon | COL | 127.2 | 124 | 3.2 |

D Girardi | NYR | 120.7 | 125 | -4.3 |

R Hainsey | WPG | 120.5 | 123 | -2.5 |

D Seidenberg | BOS | 119.8 | 115 | 4.8 |

L Smid | EDM | 114.7 | 119 | -4.3 |

B Orpik | PIT | 110.3 | 114 | -3.7 |

A MacDonald | NYI | 109.8 | 123 | -13.2 |

J Carlson | WSH | 109.4 | 123 | -13.6 |

The full paper can be found HERE.

Note: Brian’s contributions to this project, with the exception of minor edits of the paper, were made while he was an Associate Professor in the Department of Mathematical Sciences at the United States Military Academy, West Point, NY, prior to joining the Florida Panthers.

]]>*Recently there has been a good deal of interest in the use of statistical methods and statistical thinking in sports. Work of this kind has been going on for many many years though often behind the scenes. With the publication of Michael Lewis’ Moneyball and the subsequent movie of the same name, the interest in this kind of work has grown extensively. Further evidence of this can be found in the increased number of conferences devoted to the topic and the increased use of these topics in the sports media. In this talk, I will begin with an overview of sports analytics including a discussion of the general approaches and methods. Having done that, I will discuss two examples from my own recent work: Rink effects in the NHL and NHL player ratings.*

Thanks to Shirley Mills at Carleton University for the invitation to speak and for lunch. Thanks to her students for some enlightening conversations. Was also nice to see some folks I know from the Canadian Border Services Agency in the audience.

Here are the slides from that talk.

]]>