[CorsiHockeyLeague]

Originally Posted by Enoch Root

Their model may say that this player is better than that player at X (defensive zone play, for example), but they haven't, and most likely can't, demonstrate that the results actually prove this (valid output).

Okay, I am with you but waiting to hear why they haven't and most likely can't demonstrate that.

We want to know who is better defensively, Smith or Jones. The model tells us that Smith has a WAR of 18% and Jones has a WAR of 28%. Can we conclude that Jones is better defensively than Smith?

Well, no. They have a defensive WAR of some number, generally rounded to the nearest tenth. For Nurse, it's -1.6. WAR stands for Wins Above Replacement - it's a number of won games added by the player in that area. The percentages are intended to express what percentage of active players are below the player in question in that stat.

We can conclude (obviously) that Jones scored higher on the inputs that the model is using in order to try and illustrate that they are better defensively, but taking that to the next step, they can't demonstrate that Jones is a better player. We have to assume that the input scores will determine the desired conclusion in order to bother using the model.

This seems like the issue is defining a "better player". The model developer is saying, look, we have thousands and thousands of shots of data. X% of shots in this area near the slot result in goals, Y% in shots over here on the half wall result in goals, and we can see, compared to a replacement level defenseman, where players playing against Darnell Nurse take their shots from, and how many they get. We can adjust that rate by looking at where they get those shots when he's playing with teammate A, B, C, D by looking at their own individual results when playing with or without Nurse on the ice. We can do the same for opposing players, normalizing by how good they are at getting high danger shots, and the volume of them, against other competition league wide. We can then factor in things other data we have access to, such as turnovers, possession exits, completed passes out of the zone, to factor in how those affect the frequency and dangerousness of chances given up by his team when Darnell Nurse is on the ice, again adjusted according to how good his teammates are at those things and how good the competition he plays against is at securing turnovers and preventing controlled zone exits. And taken together, all of those things tell you a lot about whether Darnell Nurse is good at playing defense or not.

I gather that what you are saying is that those things can't actually tell us much that's useful about whether a player is good at defense. If so, I guess that's where we'll have to disagree. I think those things (particularly shot volume, shot location and the ability to get the puck out of the zone in transition) are the most important skills and outcomes when it comes to keeping the puck out of your team's net. I base that view on a large amount of analysis that has been done over many years by a wide variety of analysts.

Anyway, I have derailed the thread enough for one day. And this is a very challenging thing to discuss on a message board with short, two-paragraph replies. We can agree to disagree.

Sure, no worries. I do understand where your skepticism is coming from.