As mentioned in the draft thread I've been tracking draft rankings in a spreadsheet since 2013 and was trying to figure out a way to evaluate how the various experts have performed. This is what I've come up with.
1. Evaluating from 2013-2019 as I started recording rankings in 2013 and anything past 2019 seems premature (do we really know what Quinton Byfield or Alexis Lafreniere is yet?)
2. Using the first round players (so 30 or 31 players depending on the year) from each list as some rankings I don't have further than that and we mostly care about the top of the draft
3. Evaluating based on the actual results of the players in the NHL - not where they were drafted as these lists aren't meant to be mock drafts but rather are how the expert thinks the player will perform. To achieve this I decided to use the career Point Share calculation from HockeyReference.com as it allows comparison across positions. This also means that players with similar results will have similar values.
4. Weight the results based on where the player was ranked - getting a good player in your ranking at #25 is fine, but if the player you put #2 was a bust that should count against you pretty heavily. I decided the simplest way to weight the rankings was by multiplying the point shares by the inverse of the slot - so for a year where the first round had 30 picks for the #1 ranked player's point shares are multiplied by 30, the #2 player by 29, etc. I'm not convinced that the top players in the ranking shouldn't be weighted even more heavily but this is at least a start.
5. For a baseline I then calculated the same value based on the actual draft positions for the year
6. To allow comparison across years I then took each list's score as a percentage of the baseline. So if the score for the actual draft was 9279 (the 2017 score) and Bob McKenzie's list scored 9204 he would end up with 9204/9279*100 = 99.2 as his score for that year. Above 100 means the list outperformed the actual draft, below 100 is worse.
My assumption going is was that Bob McKenzie would be right around the actual GM results as his list is an average of NHL scouts. I figured Craig Button and Corey Pronman would probably vary quite a bit from year to year as they tend to go against the consensus more than most. So, how did the various evaluators do?
Code:
Overall 2019 2018 2017 2016 2015 2014 2013
Future Considerations 100.0 95.1 103.4 95.1 104.5 99.6 103.6 98.7
Bob McKenzie 99.7 100.6 102.3 99.2 100.3 99.9 101.1 94.3
Craig Button 99.3 99.2 101.5 95.4 105.4 100.4 104.8 88.7
HockeyProspect.com 99.2 99.0 102.5 97.0 99.9 103.9 93.0
McKeen's 99.0 94.6 105.7 94.9 99.2 102.1 101.2 94.9
Eldon MacDonald 98.8 95.8 100.8 99.8
Christopher Ralph 98.7 102.0 99.3 94.7
Ryan Kennedy 97.4 98.6 93.2 100.7 98.5 101.0 92.7
Last Word on Sports 97.3 94.1 99.3 90.8 103.1 99.1 97.6
Corey Pronman 96.2 100.1 86.9 85.8 99.9 95.4 104.1 101.1
ISS 95.8 96.9 98.2 89.2 98.4 99.4 93.3 95.0
Ryan Pike 95.7 94.3 86.6 106.2
Scott Wheeler 95.3 92.1 98.6
As expected, McKenzie is generally right around 100. Future Considerations had a few very good years. Button had a terrible 2013 and excellent 2014 and 2016 rankings. Pronman was awful in 2017 and 2018. ISS is not good. Wheeler didn't do well, but it was only a 2 draft sample for him. Overall the NHL GMs get better results than the expert rankings.
So what does it all mean? Probably not much but I thought some of you may find it interesting.
Evaluating Past Draft Rankings
