05-24-2024, 02:52 PM
|
#1
|
Powerplay Quarterback
|
Evaluating Past Draft Rankings
As mentioned in the draft thread I've been tracking draft rankings in a spreadsheet since 2013 and was trying to figure out a way to evaluate how the various experts have performed. This is what I've come up with.
1. Evaluating from 2013-2019 as I started recording rankings in 2013 and anything past 2019 seems premature (do we really know what Quinton Byfield or Alexis Lafreniere is yet?)
2. Using the first round players (so 30 or 31 players depending on the year) from each list as some rankings I don't have further than that and we mostly care about the top of the draft
3. Evaluating based on the actual results of the players in the NHL - not where they were drafted as these lists aren't meant to be mock drafts but rather are how the expert thinks the player will perform. To achieve this I decided to use the career Point Share calculation from HockeyReference.com as it allows comparison across positions. This also means that players with similar results will have similar values.
4. Weight the results based on where the player was ranked - getting a good player in your ranking at #25 is fine, but if the player you put #2 was a bust that should count against you pretty heavily. I decided the simplest way to weight the rankings was by multiplying the point shares by the inverse of the slot - so for a year where the first round had 30 picks for the #1 ranked player's point shares are multiplied by 30, the #2 player by 29, etc. I'm not convinced that the top players in the ranking shouldn't be weighted even more heavily but this is at least a start.
5. For a baseline I then calculated the same value based on the actual draft positions for the year
6. To allow comparison across years I then took each list's score as a percentage of the baseline. So if the score for the actual draft was 9279 (the 2017 score) and Bob McKenzie's list scored 9204 he would end up with 9204/9279*100 = 99.2 as his score for that year. Above 100 means the list outperformed the actual draft, below 100 is worse.
My assumption going is was that Bob McKenzie would be right around the actual GM results as his list is an average of NHL scouts. I figured Craig Button and Corey Pronman would probably vary quite a bit from year to year as they tend to go against the consensus more than most. So, how did the various evaluators do?
Code:
Overall 2019 2018 2017 2016 2015 2014 2013
Future Considerations 100.0 95.1 103.4 95.1 104.5 99.6 103.6 98.7
Bob McKenzie 99.7 100.6 102.3 99.2 100.3 99.9 101.1 94.3
Craig Button 99.3 99.2 101.5 95.4 105.4 100.4 104.8 88.7
HockeyProspect.com 99.2 99.0 102.5 97.0 99.9 103.9 93.0
McKeen's 99.0 94.6 105.7 94.9 99.2 102.1 101.2 94.9
Eldon MacDonald 98.8 95.8 100.8 99.8
Christopher Ralph 98.7 102.0 99.3 94.7
Ryan Kennedy 97.4 98.6 93.2 100.7 98.5 101.0 92.7
Last Word on Sports 97.3 94.1 99.3 90.8 103.1 99.1 97.6
Corey Pronman 96.2 100.1 86.9 85.8 99.9 95.4 104.1 101.1
ISS 95.8 96.9 98.2 89.2 98.4 99.4 93.3 95.0
Ryan Pike 95.7 94.3 86.6 106.2
Scott Wheeler 95.3 92.1 98.6
As expected, McKenzie is generally right around 100. Future Considerations had a few very good years. Button had a terrible 2013 and excellent 2014 and 2016 rankings. Pronman was awful in 2017 and 2018. ISS is not good. Wheeler didn't do well, but it was only a 2 draft sample for him. Overall the NHL GMs get better results than the expert rankings.
So what does it all mean? Probably not much but I thought some of you may find it interesting.
|
|
|
The Following 35 Users Say Thank You to Arsenal14 For This Useful Post:
|
1qqaaz,
apiquard,
Badgers Nose,
Bankai,
Bend it like Bourgeois,
Bingo,
bizaro86,
Bourque's Twin,
Calgary4LIfe,
CF84,
Charcot,
chummer,
cral12,
Eric Vail,
FacePaint,
Fan69,
Flames_F.T.W,
getbak,
HitterD,
I-Hate-Hulse,
icarus,
ignite09,
Jay Random,
jharp,
Jiri Hrdina,
looooob,
mile,
Reggie Dunlop,
rogermexico,
Samonadreau,
shutout,
Tbull8,
The Fonz,
transplant99,
Yrebmi
|
05-24-2024, 02:57 PM
|
#2
|
First Line Centre
|
I’d be curious to see how accurate these rankings are relative to how the draft plays out. My gut says McKenzie has the smallest variance still
|
|
|
05-24-2024, 03:02 PM
|
#3
|
Powerplay Quarterback
Join Date: Mar 2006
Location: In my office...is it 5:00 yet???
|
Thats it! i'm taking Ryan Pike's book back after that performance.
Thank you so much for putting this together. Button comes out looking a lot better than many (including me) would have thought.
|
|
|
The Following User Says Thank You to HitterD For This Useful Post:
|
|
05-24-2024, 03:07 PM
|
#4
|
Powerplay Quarterback
|
Quote:
Originally Posted by HitterD
Thats it! i'm taking Ryan Pike's book back after that performance.
Thank you so much for putting this together. Button comes out looking a lot better than many (including me) would have thought.
|
In fairness, Pike only had three drafts and had some pretty wild variance. His 2016 list fared the best of any expert in any year and then he missed hard in 2017. 2016 he hit on McAvoy, DeBrincat, Chychrun and Samuel Girard. 2017 was hurt by Nolan Patrick busting and being lower on Heiskanen and Pettersson than most plus not having Jason Robertson in the first round.
|
|
|
The Following User Says Thank You to Arsenal14 For This Useful Post:
|
|
05-24-2024, 03:28 PM
|
#5
|
Franchise Player
|
Good reminder, when does the FC draft guide come out, always enjoy that one and have appreciated the discounts on the board.
|
|
|
05-24-2024, 03:38 PM
|
#6
|
Powerplay Quarterback
Join Date: Mar 2006
Location: In my office...is it 5:00 yet???
|
Quote:
Originally Posted by Arsenal14
In fairness, Pike only had three drafts and had some pretty wild variance. His 2016 list fared the best of any expert in any year and then he missed hard in 2017. 2016 he hit on McAvoy, DeBrincat, Chychrun and Samuel Girard. 2017 was hurt by Nolan Patrick busting and being lower on Heiskanen and Pettersson than most plus not having Jason Robertson in the first round.
|
Oh for sure, nothing meant by it other than a quick rib. I'll also take the opportunity to recommend his book to any flames fans i know, its such a great read
|
|
|
05-24-2024, 04:50 PM
|
#7
|
Franchise Player
|
Awesome work
|
|
|
The Following User Says Thank You to Jiri Hrdina For This Useful Post:
|
|
05-24-2024, 05:08 PM
|
#8
|
#1 Goaltender
Join Date: Apr 2022
Location: California
|
Thanks, I love threads like this!
Couple questions:
- Did you simply rank the career point share totals or is the variation between them preserved? For example, if 1st is 40 and 2nd is 10, that is a very large difference that should be preserved in any evaluation of scouts in my opinion, as the next year 1st could be 24.4 and 2nd could be 24.2 - a trivial difference.
- Rather than 1st = 30, 2nd = 29, etc., can you try using the values listed here? https://soundofhockey.com/2022/06/06...l-draft-picks/ My feeling is that it will enhance the differences between the more critical earlier picks.
|
|
|
The Following User Says Thank You to butterfly For This Useful Post:
|
|
05-24-2024, 05:12 PM
|
#9
|
Scoring Winger
Join Date: Feb 2007
Location: Why is there oil everywhere?
|
Thanks for doing this. Must've taken you a long time. Love reading it!
|
|
|
05-24-2024, 06:20 PM
|
#10
|
Franchise Player
Join Date: Oct 2001
Location: Singapore
|
Love analyses like this, thank you!
So if we are to regard Bob McKenzie and FC as the most reliable rankings, one thing I find interesting looking at their latest lists is that they both have Konsta Helenius ahead of Tij Iginla.
This is going to be a very interesting draft as a Flames fan!
__________________
Shot down in Flames!
|
|
|
The Following 2 Users Say Thank You to icarus For This Useful Post:
|
|
05-24-2024, 07:17 PM
|
#11
|
First Line Centre
Join Date: Jul 2009
Location: Calgary
|
Love this and thanks for doing this Arsenal14. Cool to see my old stuff up there, admittedly. My pal E-Mac (Eldon) gets bragging rights in this analysis, though cool to see I was trending upwards each subsequent year. At least I have bragging rights over my pal Pike lol, but he killed it in 2016.
I would note (and will be interesting to track), E-Mac is our chief scout at Upside Hockey and is the man behind the mega spreadsheet style rankings we keep highlighting in the 2024 NHL draft thread, and of course our main man, CP's own, Sandman our prolific prospect profiler (we're over 200 profiles now).
I have to go back through to confirm, but at least one or two of the last two drafts, E-Mac actually had 1-2 more prospects ranked correctly going in the first round more than Bobby Mac did, fwiw.
|
|
|
The Following User Says Thank You to cral12 For This Useful Post:
|
|
05-25-2024, 08:55 AM
|
#12
|
Powerplay Quarterback
|
Quote:
Originally Posted by butterfly
Thanks, I love threads like this!
Couple questions:
- Did you simply rank the career point share totals or is the variation between them preserved? For example, if 1st is 40 and 2nd is 10, that is a very large difference that should be preserved in any evaluation of scouts in my opinion, as the next year 1st could be 24.4 and 2nd could be 24.2 - a trivial difference.
- Rather than 1st = 30, 2nd = 29, etc., can you try using the values listed here? https://soundofhockey.com/2022/06/06...l-draft-picks/ My feeling is that it will enhance the differences between the more critical earlier picks.
|
I used the actual point share values for the reason you mentioned. I might try changing the weights at some point to see what that does. It should be pretty quick to test.
|
|
|
The Following User Says Thank You to Arsenal14 For This Useful Post:
|
|
05-25-2024, 09:41 PM
|
#13
|
Powerplay Quarterback
|
Quote:
Originally Posted by butterfly
|
Code:
Overall 2019 2018 2017 2016 2015 2014 2013
Bob McKenzie 100.2 101.2 102.7 101.6 100.4 100.3 100.2 94.9
Christopher Ralph 99.5 102.3 99.4 96.8
Craig Button 99.4 102.8 101.0 94.5 108.0 99.1 104.1 86.2
Eldon MacDonald 99.2 98.0 101.0 98.6
Future Considerations 98.6 97.6 100.9 90.0 105.8 99.4 101.1 95.7
HockeyProspect.com 98.2 99.2 102.2 99.5 99.8 96.6 92.1
McKeen's 97.2 97.5 103.1 90.4 97.4 101.9 99.4 90.9
Last Word on Sports 96.7 97.1 97.6 83.9 106.0 98.9 97.0
Ryan Kennedy 96.5 99.7 85.5 101.2 97.9 101.7 92.9
Ryan Pike 95.5 99.5 81.0 105.9
Scott Wheeler 95.2 93.5 97.0
Corey Pronman 95.1 100.6 89.7 83.7 97.6 98.2 101.5 94.7
ISS 94.3 97.9 97.0 83.0 101.4 97.3 90.0 93.8
With this weighting it's very clear that McKenzie matches the actual first picks of the draft most years. Congrats to cral12 who moves into second place. ISS still is terrible and Craig Button remains right near the top.
The 2016 draft provided lots of room to beat the actual GM results for any expert who had Juolevi, Puljujarvi and Alexander Nylander out of the top 10.
On the other end few of the experts did well in 2017 where teams picked Heiskanen/Makar/Pettersson at 3/4/5 and pretty much no rankings came close to matching that (McKenzie was close with those players at 3/4/7).
|
|
|
The Following 3 Users Say Thank You to Arsenal14 For This Useful Post:
|
|
05-26-2024, 09:00 AM
|
#14
|
Owner
Join Date: Dec 2001
Location: Calgary
|
So interesting!
I think your adjustment gives more separation and a more interesting look.
And personally I like Button getting some love after the drive bys!
|
|
|
The Following 4 Users Say Thank You to Bingo For This Useful Post:
|
|
05-26-2024, 10:02 AM
|
#15
|
First Line Centre
Join Date: Jul 2009
Location: Calgary
|
Super intriguing, Arsenal14 -again, thx for doing and sharing.
***
FWIW, I just went back and reviewed Upside Hockey's 1st rnd rankings last 2 drafts - E-Mac did well I'd say:
2022:
https://twitter.com/user/status/1672270977840082944
2023:
Bobby Mac got 28/32, while E-Mac landed 27/32, so a few unexpected 1st rnders.
Of the 5 who E-Mac, "missed" on, all 5 were ranked in the 2nd, two of them really early 2nd rnders. He had Easton Cowan mid-2nd and higher than most any other ranking and his #1 sleeper for the draft - he's went on to have a killer draft + 1 yr.
|
|
|
05-26-2024, 11:21 AM
|
#16
|
Our Jessica Fletcher
|
Quote:
Originally Posted by Bingo
So interesting!
I think your adjustment gives more separation and a more interesting look.
And personally I like Button getting some love after the drive bys!
|
“That there folks is why Craig Button will never get another sniff of the NHL as a scout.”
|
|
|
05-26-2024, 11:51 AM
|
#17
|
GOAT!
|
Now that I know how McKenzie comes up with his list, I don't think it's fair to compare his "success" with places/people who base their lists on their own scouting.
Anyone with contacts can do what he does.
|
|
|
The Following User Says Thank You to FanIn80 For This Useful Post:
|
|
05-26-2024, 12:44 PM
|
#18
|
Owner
Join Date: Dec 2001
Location: Calgary
|
Quote:
Originally Posted by FanIn80
Now that I know how McKenzie comes up with his list, I don't think it's fair to compare his "success" with places/people who base their lists on their own scouting.
Anyone with contacts can do what he does.
|
It's not fair if you use it to call him a genius, as he's only asking questions of 10 other qualified hockey scouts.
But if you want to evaluate the McKenzie list results year after year it's very important information.
|
|
|
The Following 2 Users Say Thank You to Bingo For This Useful Post:
|
|
05-26-2024, 01:30 PM
|
#19
|
GOAT!
|
Quote:
Originally Posted by Bingo
It's not fair if you use it to call him a genius, as he's only asking questions of 10 other qualified hockey scouts.
But if you want to evaluate the McKenzie list results year after year it's very important information.
|
For sure. I'm not discounting the value of his phone book, but that's all it really is. Useful info, but it should just be referred to as "team consensus" or something.
|
|
|
The Following User Says Thank You to FanIn80 For This Useful Post:
|
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
All times are GMT -6. The time now is 08:44 AM.
|
|