Today 31 NHL front offices and some of the world’s top hockey prospects are in Chicago for the NHL Entry Draft. While some gifted 18-year-olds will realize their childhood dreams of putting on an NHL sweater, an early-round selection hardly guarantees an All-Star appearance or even a lengthy NHL career. To better understand the dynamics of the draft and the role analytics can play in informing a team’s decision-making process, we spoke with Tyler Dellow. A three-time panelist at SSAC, Dellow has served as a consultant to multiple NHL teams in addition to driving the larger discussion of hockey analytics online. In 2005 he created mc79hockey.com, a blog dedicated to exploring the use of data in hockey, that quickly became a leading site for the development and testing of analytic concepts in hockey. He currently writes for The Athletic, and his work has also appeared in sportsnet.ca and Sportsnet Magazine. You can find him on Twitter at @dellowhockey. Get more of Dellow’s insights by watching the SSAC 17 hockey panel here.
What are some of the things about the NHL draft that make it especially challenging for a front office in comparison to other sports?
There’s a mix of issues. First, hockey players tend to be drafted at a younger age than players in the NFL or NBA. As with MLB, where players become draft eligible at 18, it’s harder to predict the future path of younger athletes. Second, hockey players are spread out into an unimaginable number of leagues. Teams will draft players from major junior and Tier II junior in Canada and the United States, the NCAA, American high schools, and European senior and junior leagues. All of these leagues have different levels of play, which makes synthesizing rankings of the players particularly difficult.
Finally, there’s the nature of the sport itself. Hockey is particularly teammate dependent in a way that games like basketball and baseball aren’t. If you’re a great pitcher on a poor team, you’ll still put up a great ERA (or FIP, or whatever your preferred metric may be) and pile up strikeouts, even if you don’t get the wins you might otherwise get. It’s harder in hockey because even the results that you achieve when you’re on the ice are dependent on the players with you. Hockey has some history of disappointment when drafting the third or fourth best player from a great team as teams discover that the circumstances made the player appear better than he was.
Among the challenges are the differences among the leagues prospects play in. You have kids who play 70 games in major junior, guys who play 30 games in the NCAA, and guys who play 40-50 games in the Finnish Elite League. From an analytics perspective, how can you control for the variability in playing style and quality of competition?
This is a difficult problem because these leagues are largely self-contained. The NCAA does not play a slate of games against the Russian junior league each season, for example, that would enable us to develop some understanding of the relative strengths of the leagues.
Generally speaking, the approach is to [look at] players with a similar level of performance from that league in the past and then look at how those players performed as their careers progressed. This can be complicated by the changing quality of the league – for example, the USHL was not an important producer of NHL players twenty years ago; today it’s an important feeder league. Accordingly, you have to be constantly updating your model of the league’s strength and considering whether there’s been a shift in the league’s strength that should impact your assessment of what a player’s performance means in terms of his prospects.
What kind of statistics are helpful in evaluating prospects? You’ve done some work with the difference in GF% on and GF% off for defenseman age 20. What does that tell us about a player’s potential?
Hockey suffers from a lack of data when it comes to prospects. One of the more important analytical insights is that teams should be interested in players who positively impact goal difference. The problem with this is that goals are rare in hockey and there’s a good deal of randomness in the game. Over time, the best players have a visibly positive impact on their team’s goal difference when they’re on the ice (subject to confounding factors, like defensemen playing at an extremely high level of competition). In the short run though, that impact can be swamped by bounces.
At the NHL level, we have more data about the process that leads to goal difference – we know who is on the ice for each shot and can look to identify players who seem to be positively impacting the process that leads to goals. In the lower levels, we frequently just have goal data if we’re lucky, which has to be taken with a grain of salt because it’s entirely possible for bounces to make a player’s data look particularly good or bad over the course of a season or two.
My research, along with the research of other hockey researchers, has shown that defensemen who become high end NHL defensemen tend to be impacting the game significantly at a fairly young age. The received hockey wisdom is that defensemen don’t figure out the game until they’re in their mid-twenties. When you look at the data, this doesn’t hold up very well.
The normal course for most players involves graduating from junior hockey to the American Hockey League at age 20. As such, comparing GF% on with GF% when a player is off the ice at that stage of his career enables us to see whether a player’s team is doing better when he’s on the ice. When you combine that with the knowledge that most defensemen who end up making a difference show it fairly quickly, you can identify defensemen with a chance to help a team, even if they aren’t in the NHL yet.
You mentioned Jason Demers of the Panthers on Twitter the other day, asking how many teams missed on him (he was drafted 186th overall in the 7th round). He put up impressive numbers in the QMJHL. What are some other misses that jump out at you in recent years?
From my perspective, the biggest miss in recent years in the sense of teams passing on him is Nikita Kucherov, who was drafted by the Tampa Bay Lightning 58th overall in 2011. In this past season, Kucherov scored 40 goals and added 45 assists, finishing tied for second in goals and tied for fifth in points. It’s unusual to find truly elite offensive forwards late in the second round of the NHL draft.
In Kucherov’s case, he was likely penalized by what hockey people refer to as “the Russian Factor.” There was, at the time, a perception that Russian players were particularly risky draft picks due to the possibility that they may prefer to stay and play in the Kontinental Hockey League (KHL), which was quite financially powerful. Kucherov was the first player drafted from a Russian team in 2011.
In 2010, Vladimir Tarasenko, another Russia-based player, was drafted by St. Louis in the middle of the first round. He was picked immediately after Los Angeles selected a player named Derek Forbort. Forbort has since played 96 games. Tarasenko has scored 145 NHL goals. Los Angeles might well have another Stanley Cup or two if they could score. Tarasenko and Evgeny Kuznetsov were the only two KHL players drafted in the first five rounds in that season.
St. Louis and Tampa Bay have reaped huge rewards from their willingness to accept the uncertainty that these players would be willing to or could be extricated from Russia when the time came to do so. Intriguingly, the playing career of Tampa Bay GM Steve Yzerman was positively altered by the Detroit Red Wings being willing to take similar risks. Detroit had what is commonly recognized as one of the greatest drafts of all time in 1989, when they drafted Sergei Fedorov and Vladimir Konstantinov from the then Soviet Union in the fourth and eleventh rounds respectively. At the time, the Soviet Union limited the ability of players to leave the country – that changed quickly after communism fell. Steve Yzerman was the beneficiary of that as a player and his willingness to take a risk on players who seemed problematic to acquire paid off again with Kucherov.
This is an analytics issue because it goes to the core purpose of the draft and what type of players are hardest to acquire. There is considerable evidence that the major benefit of the draft is the opportunity to acquire stars, who are very difficult to otherwise acquire. From that perspective, taking a risk on a Kucherov or a Tarasenko, rather than making a safe choice who will play on your third line, is a no-brainer. Despite this, teams still like safe choices.
Do you see any biases that might lead teams to underrate certain types of players and overrate others?
Size continues to be the biggest issue. While there is a size point at which it doesn’t matter how skilled a player might be, he’s virtually certain not to succeed in the NHL, teams chase large defensemen and large forwards. As a result, the track records of those players compared to the players drafted around them tend to be poor. In particular, large forwards drafted after the first round have a poor track record, as do large defensemen who aren’t able to produce points in junior. With respect to large forwards, the issue is pretty simple: if there’s a realistic possibility that he can play, he’ll probably be drafted in the first round.
Every so often a team hits on one of those players – Boston drafted Milan Lucic 50th overall in 2006, coming off a season in which he scored 19 points. Lucic became a fine NHL player and dozens of draft picks have been burned ever since, trying to find the next Lucic. When someone inevitably lucks into him, dozens of draft picks will again be burned in search of the next one.
Another bias that pops up is drafting from leagues that are well scouted beyond the point at which that league generally produces players. Canadian major junior hockey is the most heavily scouted junior hockey in the world. The legitimate prospects tend to be gone within the first 100 picks or so. Despite that, teams continue drafting from Canadian major junior beyond the point at which they’d likely be better off gambling on a player from a less heavily scouted league.
At SSAC 17 the Florida Panthers discussed how they combine the work of their scouts with the work of their analytics team. From your perspective what kind of impact is analytics having on draft day and how many teams are taking an approach similar to the Panthers?
I think that the impact of analytics on hockey is small but growing. Hockey is a conservative sport, extremely resistant to change or to examining the ideas that have governed the decision making in the game for a hundred years. As such, those new ideas will need to develop a track record of success before they are broadly adopted. There have been some notable successes, although mostly in identifying prospects who were overrated by the league at large, were drafted high and quickly displayed that their potential was limited.
With that said, nobody likes to take a prospect who has been identified by a small but vocal group as a bad risk – it looks bad if he fails. This will likely be the first place in which the analytics impact is really felt, as teams perhaps shy away from prospects that have well recognized and documented red flags.
There are likely only three or four teams who have taken an approach as systematic and aggressive as that of the Panthers. If you know the players who pop out, you notice teams that seem to be picking them. If Florida or one of the other teams that takes this approach has success, more teams will likely follow.