Quote:
Originally Posted by CorsiHockeyLeague
Well, maybe I'm expressing this incorrectly. I guess what I mean is that the burden has been met, once the person who's created the model has explained how the model has been developed, how it has been tested, and how it has performed. They've done that. So now it would be up to whoever is attempting to discredit the model and claim that it isn't actually useful in the ways that the developer says it is to explain why they're wrong, and why the degree of accuracy they claim to have achieved is a mirage. As you say, you have some experience here, which would seem to suggest that you're qualified to do just that.
I obviously can't explain why you, as one person with some knowledge and expertise on statistical models, have come to the conclusion that the data available for hockey is inadequate to yield results that can be fairly used to compare one player to another, while another person with knowledge and expertise has come to the opposite conclusion. You'd have to explain where the difference of opinion lies.
I certainly see the challenge you're highlighting when talking about reliably assigning outcomes (data) to individual players. I think most hockey analytics people would agree that that's the main source of error in their predictions. But it seems to me that there are enough data points to narrow the error bars to an extent where the model output is reliable enough to be useful and worthwhile. Again, the bar to clear here is nowhere ear perfection.
Well, that's definitely true, though it isn't the fault of the person doing the analysis. If you want to tell me I'm misinterpreting their results and how I'm doing that I'm all ears.
|
This is where I disagree.
Their model may say that this player is better than that player at X (defensive zone play, for example), but they haven't, and most likely can't, demonstrate that the results actually prove this (valid output).
We want to know who is better defensively, Smith or Jones. The model tells us that Smith has a WAR of 18% and Jones has a WAR of 28%. Can we conclude that Jones is better defensively than Smith?
We can conclude (obviously) that Jones scored higher on the inputs that the model is using in order to try and illustrate that they are better defensively, but taking that to the next step, they can't demonstrate that Jones is a better player. We have to
assume that the input scores will determine the desired conclusion in order to bother using the model. But that assumption is a giant leap of faith and very difficult to demonstrate (especially with the unique challenges that hockey resents).
Anyway, I have derailed the thread enough for one day. And this is a very challenging thing to discuss on a message board with short, two-paragraph replies. We can agree to disagree.