Evaluating Past Draft Rankings

[Arsenal14]

As mentioned in the draft thread I've been tracking draft rankings in a spreadsheet since 2013 and was trying to figure out a way to evaluate how the various experts have performed. This is what I've come up with.

1. Evaluating from 2013-2019 as I started recording rankings in 2013 and anything past 2019 seems premature (do we really know what Quinton Byfield or Alexis Lafreniere is yet?)
2. Using the first round players (so 30 or 31 players depending on the year) from each list as some rankings I don't have further than that and we mostly care about the top of the draft
3. Evaluating based on the actual results of the players in the NHL - not where they were drafted as these lists aren't meant to be mock drafts but rather are how the expert thinks the player will perform. To achieve this I decided to use the career Point Share calculation from HockeyReference.com as it allows comparison across positions. This also means that players with similar results will have similar values.
4. Weight the results based on where the player was ranked - getting a good player in your ranking at #25 is fine, but if the player you put #2 was a bust that should count against you pretty heavily. I decided the simplest way to weight the rankings was by multiplying the point shares by the inverse of the slot - so for a year where the first round had 30 picks for the #1 ranked player's point shares are multiplied by 30, the #2 player by 29, etc. I'm not convinced that the top players in the ranking shouldn't be weighted even more heavily but this is at least a start.
5. For a baseline I then calculated the same value based on the actual draft positions for the year
6. To allow comparison across years I then took each list's score as a percentage of the baseline. So if the score for the actual draft was 9279 (the 2017 score) and Bob McKenzie's list scored 9204 he would end up with 9204/9279*100 = 99.2 as his score for that year. Above 100 means the list outperformed the actual draft, below 100 is worse.

My assumption going is was that Bob McKenzie would be right around the actual GM results as his list is an average of NHL scouts. I figured Craig Button and Corey Pronman would probably vary quite a bit from year to year as they tend to go against the consensus more than most. So, how did the various evaluators do?

                        Overall	2019	2018	2017	2016	2015	2014	2013
Future Considerations	100.0	95.1	103.4	95.1	104.5	99.6	103.6	98.7
Bob McKenzie		99.7	100.6	102.3	99.2	100.3	99.9	101.1	94.3
Craig Button		99.3	99.2	101.5	95.4	105.4	100.4	104.8	88.7
HockeyProspect.com	99.2	99.0	102.5	97.0	99.9		103.9	93.0
McKeen's		99.0	94.6	105.7	94.9	99.2	102.1	101.2	94.9
Eldon MacDonald		98.8				95.8	100.8	99.8	
Christopher Ralph	98.7				102.0	99.3	94.7
Ryan Kennedy		97.4	98.6		93.2	100.7	98.5	101.0	92.7
Last Word on Sports	97.3	94.1	99.3	90.8	103.1	99.1	97.6	
Corey Pronman		96.2	100.1	86.9	85.8	99.9	95.4	104.1	101.1
ISS			95.8	96.9	98.2	89.2	98.4	99.4	93.3	95.0
Ryan Pike		95.7	94.3		86.6	106.2
Scott Wheeler		95.3	92.1	98.6

As expected, McKenzie is generally right around 100. Future Considerations had a few very good years. Button had a terrible 2013 and excellent 2014 and 2016 rankings. Pronman was awful in 2017 and 2018. ISS is not good. Wheeler didn't do well, but it was only a 2 draft sample for him. Overall the NHL GMs get better results than the expert rankings.

So what does it all mean? Probably not much but I thought some of you may find it interesting.

[Tbull8]

I’d be curious to see how accurate these rankings are relative to how the draft plays out. My gut says McKenzie has the smallest variance still

[HitterD]

Thats it! i'm taking Ryan Pike's book back after that performance.

Thank you so much for putting this together. Button comes out looking a lot better than many (including me) would have thought.

[Arsenal14]

Originally Posted by HitterD

Thats it! i'm taking Ryan Pike's book back after that performance.

Thank you so much for putting this together. Button comes out looking a lot better than many (including me) would have thought.

In fairness, Pike only had three drafts and had some pretty wild variance. His 2016 list fared the best of any expert in any year and then he missed hard in 2017. 2016 he hit on McAvoy, DeBrincat, Chychrun and Samuel Girard. 2017 was hurt by Nolan Patrick busting and being lower on Heiskanen and Pettersson than most plus not having Jason Robertson in the first round.

[Matty81]

Good reminder, when does the FC draft guide come out, always enjoy that one and have appreciated the discounts on the board.

[HitterD]

Originally Posted by Arsenal14

In fairness, Pike only had three drafts and had some pretty wild variance. His 2016 list fared the best of any expert in any year and then he missed hard in 2017. 2016 he hit on McAvoy, DeBrincat, Chychrun and Samuel Girard. 2017 was hurt by Nolan Patrick busting and being lower on Heiskanen and Pettersson than most plus not having Jason Robertson in the first round.

Oh for sure, nothing meant by it other than a quick rib. I'll also take the opportunity to recommend his book to any flames fans i know, its such a great read

[Jiri Hrdina]

Awesome work

[butterfly]

Thanks, I love threads like this!

Couple questions:

- Did you simply rank the career point share totals or is the variation between them preserved? For example, if 1st is 40 and 2nd is 10, that is a very large difference that should be preserved in any evaluation of scouts in my opinion, as the next year 1st could be 24.4 and 2nd could be 24.2 - a trivial difference.

- Rather than 1st = 30, 2nd = 29, etc., can you try using the values listed here? https://soundofhockey.com/2022/06/06...l-draft-picks/ My feeling is that it will enhance the differences between the more critical earlier picks.

[Bankai]

Thanks for doing this. Must've taken you a long time. Love reading it!

[icarus]

Love analyses like this, thank you!

So if we are to regard Bob McKenzie and FC as the most reliable rankings, one thing I find interesting looking at their latest lists is that they both have Konsta Helenius ahead of Tij Iginla.

This is going to be a very interesting draft as a Flames fan!

[cral12]

Love this and thanks for doing this Arsenal14. Cool to see my old stuff up there, admittedly. My pal E-Mac (Eldon) gets bragging rights in this analysis, though cool to see I was trending upwards each subsequent year. At least I have bragging rights over my pal Pike lol, but he killed it in 2016.

I would note (and will be interesting to track), E-Mac is our chief scout at Upside Hockey and is the man behind the mega spreadsheet style rankings we keep highlighting in the 2024 NHL draft thread, and of course our main man, CP's own, Sandman our prolific prospect profiler (we're over 200 profiles now).

I have to go back through to confirm, but at least one or two of the last two drafts, E-Mac actually had 1-2 more prospects ranked correctly going in the first round more than Bobby Mac did, fwiw.

[Arsenal14]

Originally Posted by butterfly

Thanks, I love threads like this!

Couple questions:

- Did you simply rank the career point share totals or is the variation between them preserved? For example, if 1st is 40 and 2nd is 10, that is a very large difference that should be preserved in any evaluation of scouts in my opinion, as the next year 1st could be 24.4 and 2nd could be 24.2 - a trivial difference.

- Rather than 1st = 30, 2nd = 29, etc., can you try using the values listed here? https://soundofhockey.com/2022/06/06...l-draft-picks/ My feeling is that it will enhance the differences between the more critical earlier picks.

I used the actual point share values for the reason you mentioned. I might try changing the weights at some point to see what that does. It should be pretty quick to test.

[Arsenal14]

Originally Posted by butterfly

- Rather than 1st = 30, 2nd = 29, etc., can you try using the values listed here? https://soundofhockey.com/2022/06/06...l-draft-picks/ My feeling is that it will enhance the differences between the more critical earlier picks.

                        Overall	2019	2018	2017	2016	2015	2014	2013
Bob McKenzie		100.2	101.2	102.7	101.6	100.4	100.3	100.2	94.9
Christopher Ralph	99.5				102.3	99.4	96.8
Craig Button		99.4	102.8	101.0	94.5	108.0	99.1	104.1	86.2
Eldon MacDonald		99.2				98.0	101.0	98.6	
Future Considerations	98.6	97.6	100.9	90.0	105.8	99.4	101.1	95.7
HockeyProspect.com	98.2	99.2	102.2	99.5	99.8		96.6	92.1
McKeen's		97.2	97.5	103.1	90.4	97.4	101.9	99.4	90.9
Last Word on Sports	96.7	97.1	97.6	83.9	106.0	98.9	97.0	
Ryan Kennedy		96.5	99.7		85.5	101.2	97.9	101.7	92.9
Ryan Pike		95.5	99.5		81.0	105.9
Scott Wheeler		95.2	93.5	97.0
Corey Pronman		95.1	100.6	89.7	83.7	97.6	98.2	101.5	94.7
ISS			94.3	97.9	97.0	83.0	101.4	97.3	90.0	93.8

With this weighting it's very clear that McKenzie matches the actual first picks of the draft most years. Congrats to cral12 who moves into second place. ISS still is terrible and Craig Button remains right near the top.

The 2016 draft provided lots of room to beat the actual GM results for any expert who had Juolevi, Puljujarvi and Alexander Nylander out of the top 10.

On the other end few of the experts did well in 2017 where teams picked Heiskanen/Makar/Pettersson at 3/4/5 and pretty much no rankings came close to matching that (McKenzie was close with those players at 3/4/7).

[Bingo]

So interesting!

I think your adjustment gives more separation and a more interesting look.

And personally I like Button getting some love after the drive bys!

[cral12]

Super intriguing, Arsenal14 -again, thx for doing and sharing.
***
FWIW, I just went back and reviewed Upside Hockey's 1st rnd rankings last 2 drafts - E-Mac did well I'd say:

2022:

https://twitter.com/user/status/1672270977840082944

2023:
Bobby Mac got 28/32, while E-Mac landed 27/32, so a few unexpected 1st rnders.
Of the 5 who E-Mac, "missed" on, all 5 were ranked in the 2nd, two of them really early 2nd rnders. He had Easton Cowan mid-2nd and higher than most any other ranking and his #1 sleeper for the draft - he's went on to have a killer draft + 1 yr.

[The Fonz]

Originally Posted by Bingo

So interesting!

I think your adjustment gives more separation and a more interesting look.

And personally I like Button getting some love after the drive bys!

“That there folks is why Craig Button will never get another sniff of the NHL as a scout.”

[FanIn80]

Now that I know how McKenzie comes up with his list, I don't think it's fair to compare his "success" with places/people who base their lists on their own scouting.

Anyone with contacts can do what he does.

[Bingo]

Originally Posted by FanIn80

Now that I know how McKenzie comes up with his list, I don't think it's fair to compare his "success" with places/people who base their lists on their own scouting.

Anyone with contacts can do what he does.

It's not fair if you use it to call him a genius, as he's only asking questions of 10 other qualified hockey scouts.

But if you want to evaluate the McKenzie list results year after year it's very important information.

[FanIn80]

Originally Posted by Bingo

It's not fair if you use it to call him a genius, as he's only asking questions of 10 other qualified hockey scouts.

But if you want to evaluate the McKenzie list results year after year it's very important information.

For sure. I'm not discounting the value of his phone book, but that's all it really is. Useful info, but it should just be referred to as "team consensus" or something.

[cral12]

Thought I'd post this in this thread as well:
Interesting analysis someone shared with me. Upside faired well in this way too early look...

Analyzing the results of the 2025 NHL Entry Draft
https://nhldraft.substack.com/p/anal...ts-of-the-2025