Q&A With Athletic Calgary's Kent Wilson on Flame's Analytics

[Metro Gnome]

Originally Posted by Calgary4LIfe

I have definitely heard that they are looking at possession as being critical (and that was a point that Treliving raised when Hartley was dismissed), but I do remember his speaking about CORSI and stating that they don't use CORSI in their metrics, and instead utilize something that is more accurate. Is this a recent change, or was Treliving just being coy? I wish I could find his quote, but I am pretty sure it came from a Fan960 interview a couple of seasons back. Not questioning you, just asking for some clarification.

There's no doubt they buttress corsi / publicly available stats with in-house stuff that we are not privy too. My guess, given what I've seen come out of other clubs, is some kind of scoring chance stat.

[Metro Gnome]

Originally Posted by nobles_point

Just to sum up the Modified Areal Unit Problem. It seems to me that this is what the Oilers are doing with Kris Russel & their "own internal stats".

This is very interesting and something I'd have to take a deeper look at.

As for the Russell example in particular, I can't say if this is an issue or not, but one of the biggest risks for internal analytics is for them to turn into giant confirmation bias engines. Meaning, instead of being tools for critical analysis, they become methods to find data that essentially confirms whatever it is the decision maker is looking for.

With any of this stuff, the question to ask is always "how predictive are out stats/model? What are our assumptions and are they helping us improve our success rate?"

It sounds simple, but it's a huge challenge in a rigidly top-down culture like hockey.

[Metro Gnome]

Originally Posted by Cecil Terwilliger

Jesus people, this is why we can't have nice things. If you don't want to pay, leave. There are lots of things you have to pay for that are discussed on this forum, including the tv broadcast to tonights game. I don't hear people whining that SN isn't free.

You aren't required to pay to ask a question of the guy.

/rant


As for the thread, I'm very curious to hear what Kent has to say on this topic and how analytics have changed the way teams evaluate and put together their rosters.

I don't know if we can say there's been a big change in how teams put together rosters just yet. I don't think we've seen a sweeping change or revolution on the ice. That said, years ago when I was relatively new to writing about the game I more or less anticipated the end of the enforcer as a role in the NHL and the reduction of goalies picked in the first round.

We have also kind of seen the end of the big, hulking "defensive defender" and the rise of the two-way, transition defender. Whether this is due to natural evolution in the game or spurred by analytics is hard to say.

For me, I'm interested to see if a few things pop up in the next few years, including trying 4th lines made up of specialists (shoot out guys, PP guys, PK guys) rather than pure grinders. From a numbers perspective, there are potential gains in goal differential by going that route. The Culture and norms run counter to this, however.

Also, aging curves suggest goalies actually peak much earlier than suspected (22-23). Will teams start to try out high-quality netminders earlier? Waiting for a guy to get to 25-26 suggests clubs might be wasting peak seasons of the highest impact position.

[Anduril]

I'm always interested in the different pictures painted by various bits of analysis. Unfortunately this isn't the case for a wide variety of fans and the pseudo debate of "old-school vs statistics".

What are some things you would like to see in making statistics and analysis more widely 'accepted'. Not necessarily as fact but not immediately dismissive of it.

The NHL has started to provide more information in part of the fan interest and demand. It would be great if this trend continues so that the NHL sees the opportunity to provide better tools for measurement.

[Boreal]

Originally Posted by Metro Gnome

With any of this stuff, the question to ask is always "how predictive are out stats/model? What are our assumptions and are they helping us improve our success rate?"

It sounds simple, but it's a huge challenge in a rigidly top-down culture like hockey.

Totally agree, unconscious biases turn into answers looking for a question.

Analytics are only as good as the questions they ask. Better questions enable better answers. People are also frequently asked unaware of the difference between descriptive, predictive, & prescriptive data analysis . Fans asks questions like who is better or who will win while teams are going deeper into why a player or team does well or doesn't and when.

[Metro Gnome]

Originally Posted by thymebalm

The movement of paywalling hockey content does impact me negatively.

I've contributed my comments and feedback to many of Kent's articles over his tenure at FN - since he's moved behind the paywall I haven't been able to interact on that level with a writer for a team I'm a fan of.

Now I'm being asked to comment again, but this time I have to pay him first.

I thought it was worth pointing out.

I understand your frustration. I'll note for now that I probably wouldn't be writing this season but for the Athletic's offer.

I've spent a lot of time writing about hockey and the Flames. Thousands of hours is my guess. The Snow interview, in particular, took me over a month to arrange, several hours to plan, a couple of days to transcribe (6,000+ words) and half a day to write. Plus a couple of rounds of editing. You can see why I wouldn't have been motivated (or even able) to do that for FN.

As someone who has done this for awhile and also helped run a sports content website from a business perspective, I am keenly interested to see if we can make the pay model work at the Athletic. The goal is to have a price point that is high enough to support the best efforts of the writers involved, but not too high for the reader. It's a challenge, but an exciting one.

[corporatejay]

Originally Posted by Metro Gnome

I understand your frustration. I'll note for now that I probably wouldn't be writing this season but for the Athletic's offer.

I've spent a lot of time writing about hockey and the Flames. Thousands of hours is my guess. The Snow interview, in particular, took me over a month to arrange, several hours to plan, a couple of days to transcribe (6,000+ words) and half a day to write. Plus a couple of rounds of editing. You can see why I wouldn't have been motivated (or even able) to do that for FN.

As someone who has done this for awhile and also helped run a sports content website from a business perspective, I am keenly interested to see if we can make the pay model work at the Athletic. The goal is to have a price point that is high enough to support the best efforts of the writers involved, but not too high for the reader. It's a challenge, but an exciting one.

I got the first year for 36 bucks. That's literally 10 cents a day, something someone wouldn't even bend down to pick up off the street. I think people should support the idea and hopefully by year two there's a full staff.

[Metro Gnome]

Originally Posted by Anduril

I'm always interested in the different pictures painted by various bits of analysis. Unfortunately this isn't the case for a wide variety of fans and the pseudo debate of "old-school vs statistics".

What are some things you would like to see in making statistics and analysis more widely 'accepted'. Not necessarily as fact but not immediately dismissive of it.

The NHL has started to provide more information in part of the fan interest and demand. It would be great if this trend continues so that the NHL sees the opportunity to provide better tools for measurement.

I agree the old school vs new school battle is tiresome. to some degree, there will always be pushback from a section of sports fans, if only because people's brains aren't naturally numerate and you don't really need stats to enjoy sports. To many, it's an annoying distraction at best (and fair enough).

Chris talked about this in fact. Because he needs to get buy-in from his stakeholders the same way a writer needs to get buy-in from the audience. To that end, the way to get stats more accepted is to continue to make them as accessible as possible through clarity of explanation and tying them to clear narratives and examples. It's also about establishing an amiable dialogue/back-and-forth rather than partisan sniping between factions.

[Metro Gnome]

Originally Posted by nobles_point

Totally agree, unconscious biases turn into answers looking for a question.

Analytics are only as good as the questions they ask. Better questions enable better answers. People are also frequently asked unaware of the difference between descriptive, predictive, & prescriptive data analysis . Fans asks questions like who is better or who will win while teams are going deeper into why a player or team does well or doesn't and when.

Well said. This is a "big data" issue you see surfacing outside of sports and hockey in the business world. In marketing, for instance, the battle to demarcate between descriptive, predictive, and prescriptive analytics is on-going (and will likely be aided y AI in the near future). I have made suggestions to front offices in the past to read books on these topics, including Winning With Data and Superforecasters.

[Metro Gnome]

Thanks for the challenging questions everyone. We went a bit farther afield than I thought given the initial topic, but it was engaging. Hopefully it was interesting.

[Five-hole]

One question I have (hopefully I'm not too late) is whether analytics factor into coaching decisions. For example, if analytics shows that the team you're playing is really strong at breaking out on the right side, does the analytics department alert the coaching staff?

In theory, the coaching staff's eye test should reveal the same thing, and this is admittedly a pretty clumsy example. But perhaps like, analytics showing that a team does much poorer when player X has a bad game outletting the puck, but does very well when that player outlets the puck well. Could this inform a decision to key on that player?

[Metro Gnome]

Originally Posted by Five-hole

One question I have (hopefully I'm not too late) is whether analytics factor into coaching decisions. For example, if analytics shows that the team you're playing is really strong at breaking out on the right side, does the analytics department alert the coaching staff?

In theory, the coaching staff's eye test should reveal the same thing, and this is admittedly a pretty clumsy example. But perhaps like, analytics showing that a team does much poorer when player X has a bad game outletting the puck, but does very well when that player outlets the puck well. Could this inform a decision to key on that player?

That's a good question. I asked Chris if his department concentrates on one of three areas: player/team evaluation, on-ice tactics, and the draft. He said they look at all three areas and try to bring analysis to all of them, so that would include coaching tactics.

Of course, he stressed he works in collaboration with the coaching staff to ensure he understands what they are doing/why they are making the decisions they are before weighing in with his recommendations.

[The Cobra]

Originally Posted by Caged Great

It doesn't take into account players who do certain jobs. Take Hamonic. He's a primarily defensive minded defender and his job is to contain and limit the other team's generating offense. His job isn't to create offense. Yet according to the advanced stats, he's mediocre.

But if he does his job, and deceases the other teams offensive chances, shouldn't that result in his own team's increased offensive chances?

Now I understand if he always gest defensive zone starts, that's a difficult assignment, but analytics takes that into account.

At the end of the day, hockey is about creating offense for your team, and decreasing the offence of your opponent.

Being good at only one of those things probably won't make you a star. Analytics is likely responsible for identifying players who are only good at scoring and very poor defensively.

[Metro Gnome]

Originally Posted by The Cobra

But if he does his job, and deceases the other teams offensive chances, shouldn't that result in his own team's increased offensive chances?

Now I understand if he always gest defensive zone starts, that's a difficult assignment, but analytics takes that into account.

At the end of the day, hockey is about creating offense for your team, and decreasing the offence of your opponent.

Being good at only one of those things probably won't make you a star. Analytics is likely responsible for identifying players who are only good at scoring and very poor defensively.

Actually shot based analytics can be parsed between shot generation and shot suppression. There are players who are good at one but not the other, while the best are good at both. Mark Giordano, for instance, promotes shot generation for the Flames and suppresses shots against while he's on the ice at better than average levels. We will get even closer to assessing abilities in both directions as we develop expected goals models (for and against).

Hamonic had pretty good analytics for a few years before his injury-plagued season last year. That said, one thing that frequently lowered his shot metrics was being paired with really lousy partners against really high-end competition. No matter how good a guy is, his results are going to be influenced by the quality of his linemates and competition, especially at the extremes.

[SebC]

Originally Posted by Caged Great

I for one have been skeptical about advanced stats by in large due to their incompleteness.

It doesn't take into account players who do certain jobs. Take Hamonic. He's a primarily defensive minded defender and his job is to contain and limit the other team's generating offense. His job isn't to create offense. Yet according to the advanced stats, he's mediocre.

There's a point where, after you've tried hard enough, that if you can't measure something, then it doesn't exist. In many cases, analytics can discredit the notion of a defenceman who doesn't score being a "defensive defenceman" more than they discredit analytics.

Originally Posted by Fighting Banana Slug

I interpret the poll question to read, "if you could only pick one way to evaluate a hockey player, do you watch the games or compare advanced stats". It has to be eye test.

When you watch the games, your brain is still aggregating data. Advanced stats merely corrects for flaws in how your brain does that and allows you to fill gaps you missed. To me, a more interesting way of framing the question would be if you had to choose between watching a smaller number of games vs. data from a larger number of games. Then the stats would give you access to data that watching the games doesn't give you.

[Caged Great]

Originally Posted by Metro Gnome

Actually shot based analytics can be parsed between shot generation and shot suppression. There are players who are good at one but not the other, while the best are good at both. Mark Giordano, for instance, promotes shot generation for the Flames and suppresses shots against while he's on the ice at better than average levels. We will get even closer to assessing abilities in both directions as we develop expected goals models (for and against).

Hamonic had pretty good analytics for a few years before his injury-plagued season last year. That said, one thing that frequently lowered his shot metrics was being paired with really lousy partners against really high-end competition. No matter how good a guy is, his results are going to be influenced by the quality of his linemates and competition, especially at the extremes.

Another thing that as far as I know isn't being tracked is the mitigation of shot danger by defenders. Not all shots are created equal and there is a difference between a shot in the home plate area and some corker from the side boards. A players corsi numbers only reflect the raw data based on shots for and against, but doesn't go that added step in characterizing the quality of surrendered shots. I think that's the part where a defensive D-man of quality may have advanced stats backing them up.

The main thing I have been waiting for is RFID chips being put into jerseys to track players on ice movements. That I believe will help with understanding some of the other things more fully.

[Tyler]

Kent - is your analytic background all self-taught, or have you take any formal/related education?

Really fascinated with this stuff.

[Caged Great]

Originally Posted by SebC

There's a point where, after you've tried hard enough, that if you can't measure something, then it doesn't exist. In many cases, analytics can discredit the notion of a defenceman who doesn't score being a "defensive defenceman" more than they discredit analytics.

When you watch the games, your brain is still aggregating data. Advanced stats merely corrects for flaws in how your brain does that and allows you to fill gaps you missed. To me, a more interesting way of framing the question would be if you had to choose between watching a smaller number of games vs. data from a larger number of games. Then the stats would give you access to data that watching the games doesn't give you.

You're basically arguing that the current set of data is fine enough to sort out everything, where all I'm saying is that it doesn't go far enough. Specifically, what I am wondering about is the lack of data as far as I know about the mitigation of the danger of the shots against that would indicate the quality of the defense played by the defender. Not all shots are the same yet a good portion of the analytics treat them the same and that's limited because you're lumping all things as being the same. It's like BABIP in baseball and using that as the defining thing to characterizing a hitter's value. It doesn't account for a singles hitter like Ichiro getting on base a lot due to his speed vs a guy like Encarnacion who hammers balls into the upper deck. There's a difference between the two that isn't accounted for in the stat and why there are so many other stats to choose from.

There is also the intrinsic feel for things that some people possess. As an interior designer, decorating spaces comes naturally as I can see what's right and wrong with a space instantly and the brain starts clicking on how best to fix things almost immediately. Most people get lost along the way (they might know something doesn't look right but have no idea how to get to the "make it work" part of things).

For me, it's very much the same thing. When I'm watching a game or a player, certain things jump out at me as either being wrong or inefficient. This is more easy for evaluating prospects as it's more looking at what are they doing right than the other way around. When I look at the advanced stats, usually those back up my opinions because a guy like Backlund for example knows what he's doing and rarely makes bad decisions. Other players like Colborne you can tell that he's getting by due to his size and skill more than his brain. If Colborne's brain matched his talent level he would be a core player on Boston and wouldn't have been moved around so often. Then again, Anders Eriksson would have been a hall of famer...

They do go hand in hand though (stats and eye test).

[SebC]

Originally Posted by Caged Great

You're basically arguing that the current set of data is fine enough to sort out everything, where all I'm saying is that it doesn't go far enough. Specifically, what I am wondering about is the lack of data as far as I know about the mitigation of the danger of the shots against that would indicate the quality of the defense played by the defender. Not all shots are the same yet a good portion of the analytics treat them the same and that's limited because you're lumping all things as being the same.

With a large enough sample size (infeasibly so), we we only care if a player wins Stanley Cups. If we don't have that sample size, we care if the players wins games, because games won predicts Stanley Cup wins. If we don't have that, we care if the player scores/prevents goals, because goal differential predicts win rate. If that sample isn't big enough, we care about shots.

Which brings us to this chart.



Here, we're looking at goals vs, shots, i.e. save percentage (with some adjustments). But we're also looking at four seasons. And the correlation between a defenseman's effect on save percentage over two seasons, and his effect on save percentage on the next two seasons is very small.

Now, we could upgrade the model by replacing save percentage with an expected save percentage based on shot location, shot type (rebound, one-timer etc.), etc. that predicts save percentage better that actual save percentage, but because the sample size is already so large the gains from doing so should be expected to be minimal. As such, the correlation might strengthen, but not by much. And so, a defenceman's effect on shot quality mitigation can already be known to be of minimal importance.

https://hockey-graphs.com/2016/03/09...visited-again/

Save
Save
Save

[GranteedEV]

Re: the poll.

Honestly, I don't think eye testis the right answer, but I also think "Both" is often nothing more than the copout answer that suggests you're only willing to use analytics if they confirm your pre-existing biases. A team that signed guys like Tanner Glass, Matt Bartkowski, Troy Brouwer, and thinks they're good is the kind of team that says "both"... analytics when convenient but throw them out the window when inconvenient. At a fan level, it's worse with even more use. Fans will pull Michael Frolik's corsi to pump him up, but create narratives about how another guy's role, let's say Matt Bartkowski doesn't need good corsi as long as that player is "solid in his own zone". Narratives rule.

Should off-ice factors negate on-ice factors? Maybe character matters, I'm not arguing against character. I am arguing against "just" character to "get a good mix in the locker room".

Eye Test alone is actually IMO a better answer than a standard and convenient "both". A strict and thorough Eye Test by the right people could probably tell you more about a player than any stats, even ones as obvious as "points". But the key is to actually have watched the player and further to that to be looking for the right things. Not watch the game and then notice the "exciting" events. So eye test has to be vetted heavily with an emphasis on "is this a winning play? Or is it an entertaining play that we think wins?" So by whom the eye test is being performed is very significant. There are a lot of smart people who have made some downright incorrect eye test evaluations on established veteran players before, and are willing to die on the hill of their assessments. And there are some people who I think are the best talent evaluators in the world... Bob Hartley is one of them... but I'd still be terrified about handing him the keys to a team to go and win.

Statistics are actually a better answer than eye test in my opinion but they probably require even more thorough vetting. It actually takes some bravery to admit your eyes can fool you and to admit that authority figures in high paying positions could be prone to biases of their own. Analytics doesn't filter out excuses for the Karl Alzners of the world. At its most basic it's just the events on the ice. It's not just corsi, in fact "corsi" is trending towards the middle leaguewise. It can be something as straightforward as "Time-on-Ice". But it's also micro statistics like carry-in rates and stretch passes... it's expected goals metrics... it's being able to separate special teams from even strength. And yes, it needs to be interpreted which introduces its own biases. It's NOT perfect and it's NOT complete. It's not luck-proof. The data available is even worse at the lower levels - AHL, CHL, Europe.

What's really the "best" way to evaluate players or teams? I don't know the exact answer but I'm willing trust a chart that tells me a player gets outchanced all the time playing soft minutes over my eyes telling me a player won some board battles and made smart chips up the ice and that resulted in a goal the very same shift. Do I still trust my eyes? It's fun to do so, and for entertainment there's nothing wrong with that, but that doesn't mean I'm not prone to human error in judgement.

I know there are some really good, even great X's and O's guys on these boards and elsewhere, that can break the game down really well. But that doesn't make them without bias either. People aren't networks of logic gates.

Overall, as fans we can only trust which way our gut leans with the information we have at our disposal. But there's no excuse for management or coaches to make decisions that fans can immediately identify as bad. The "good" is the part where it gets difficult. I'm actually of the opinion that we're at a point in pro hockey where "luck" outweighs "good" anyways. It's not the NBA. Statistics can probably tell us who isn't good more clearly than who is good.

EDIT: Whoa, the above is all disorganized rambling. Meh. Sorry.