Pulling the Goalie: Hockey and Investment Implications

[CaptainYooh]

Well, the season is over for Calgary Flames; so anything hockey-related should be a fair game here now.

I thought the article below titled Pulling the Goalie: Hockey and Investment Implications could generate an entertaining and interesting discussion.

Preamble:
For those who do not know anything about hedge funds, just a very brief (and overly simplistic) background: a hedge fund is a private investment vehicle that is intended to make rich people more rich. Hedge funds invest their wealthy clients money by trying to recognize market trends and betting for and against various securities. Some obvious examples: they actively buy both long stocks (that they think would go up) and short stocks (that they think will go down). They also buy and short a vast amounts of options on bonds, currency contracts, derivatives and other financial instruments on the same principle. Modern hedge funds are managed mostly by very high-end technical people using super-complicated mathematical models recognizing market rends and sending buy/sell signals etc. Even though there have been some spectacular hedge fund failures (LTCM etc.) and even though there are hundreds of copycat funds that do not do so well, the most successful hedge funds (e.g. Medallion, Tiger) have proven that they can consistently outperform the general market and make more money (in exchange for huge fees 20%+++).

Cliff Asness is one of today's most respected hedge fund directors. He had sent this letter in the link below to the investors in March.

Summary:

In drawing parallels between professional hockey and investment management, Asness and his team have analyzed the probabilities of success and failure in NHL arising from pulling a goalie when down 1, 2, 3, 4 and 5 goals. The results are quite stunning. I found the following conclusions to be the most striking:

1. All NHL coaches do not pull goalie when a goalie SHOULD be pulled for maximum probability of success.

2. All NHL coaches are groomed to prefer losing while making proven decisions than winning while making unproven decision.

If you do not care much about reading the math and stats stuff, jump to the Chart on Page 5.

The crossover point comes at 5:40 remaining. So, at 5:50 you should not pull the goalie, but at 5:40 you should. A team that practices optimal goalie pulling gains an average of 0.02 more points per game. That is worth 1.76 points in an 82-game season, over a team that never pulls the goalie. In the 2015-16 season, 24 of the 30 NHL teams were closer than 1.76 points to either the team ahead of them in the standings or the team behind them, so this is a material difference in expected performance that comes without extra cost or work.

When down two goals, it pays to pull the goalie with 11:40 to go, less than halfway through the third period. If you score to make it one goal down, you replace your goalie until 5:40, as any earlier is too aggressive when only down by one. So, if you’re still down by one goal at 5:40, you pull again.

When down three goals, you should pull at 17:50, just over two minutes into the third period; down four goals you should pull with 35:50 remaining; and down five goals or more, you can pull at any time. These numbers may sound silly, but they make sense and matter. A team that is down five goals early in period (we might argue in this case the goalie isn’t doing any good anyway, but ignore that) still expects to collect 0.04 points on average, and can increase that 75% to 0.07 points by pulling the goalie immediately and keeping him out until the team pulls within four goals, and using the optimal rules thereafter. The intuition is the same as pulling near the end of the game. If you do nothing down five goals early your chance of winning is tiny and you have little to lose trying to climb back into the game (losing by six is not that much worse than losing by five). Though we would admit the assumptions and simplifications of our model are probably being pushed much harder for this losing-by-a-ton-early analysis.

Two reasons have been advanced for this failure to act. First, coaches are not actually rewarded for winning. They are rewarded for being perceived as good coaches. Obviously, the two are closely related but not exactly the same thing. If a basketball coach gets his team to execute crisp offensive plays with few turnovers that lead to two point baskets on 50% of possessions, he is deemed an excellent coach. If his team still loses 100 – 102, well, his players just weren’t quite good enough. If the same coach encourages his team to run-and-gun threes, with lots of turnovers and misses, but scoring on 35% of possessions, he has clearly lost control of his team. If they win 105 – 102 it’s perceived as just luck as everyone knows three point shots are risky. Essentially winning ugly is undervalued versus losing elegantly; and losing ugly can be career suicide. Once again, the way you measure risk matters in making the optimal decision.

Discussion:

On the second point, I find myself totally agreeing with the theory behind coaching to lose. It does make a lot of sense to me and I see many parallels to this same approach in both large private businesses and government organizations.

On the first point, I find it very hard to agree to the theory behind pulling goalies earlier. Remember, these are extremely intelligent people behind these calculations and there are no reasons to believe that their probability estimates are incorrect. After all, all of their calculations are not theoretical - they are based on the actual NHL production numbers - all of the numbers, in fact. I do think there is something shaky and missing though. It could be that the players would take more shots on empty net resulting in more goals. But they have not so far in the time they have...

Anyway, here is the link. The letter is not very long. Enjoy!

[edslunch]

Originally Posted by CaptainYooh

Well, the season is over for Calgary Flames; so anything hockey-related should be a fair game here now.



I thought the article below titled Pulling the Goalie: Hockey and Investment Implications could generate an entertaining and interesting discussion.



Preamble:

For those who do not know anything about hedge funds, just a very brief (and overly simplistic) background: a hedge fund is a private investment vehicle that is intended to make rich people more rich. Hedge funds invest their wealthy clients money by trying to recognize market trends and betting for and against various securities. Some obvious examples: they actively buy both long stocks (that they think would go up) and short stocks (that they think will go down). They also buy and short a vast amounts of options on bonds, currency contracts, derivatives and other financial instruments on the same principle. Modern hedge funds are managed mostly by very high-end technical people using super-complicated mathematical models recognizing market rends and sending buy/sell signals etc. Even though there have been some spectacular hedge fund failures (LTCM etc.) and even though there are hundreds of copycat funds that do not do so well, the most successful hedge funds (e.g. Medallion, Tiger) have proven that they can consistently outperform the general market and make more money (in exchange for huge fees 20%+++).



Cliff Asness is one of today's most respected hedge fund directors. He had sent this letter in the link below to the investors in March.



Summary:



In drawing parallels between professional hockey and investment management, Asness and his team have analyzed the probabilities of success and failure in NHL arising from pulling a goalie when down 1, 2, 3, 4 and 5 goals. The results are quite stunning. I found the following conclusions to be the most striking:



1. All NHL coaches do not pull goalie when a goalie SHOULD be pulled for maximum probability of success.



2. All NHL coaches are groomed to prefer losing while making proven decisions than winning while making unproven decision.



If you do not care much about reading the math and stats stuff, jump to the Chart on Page 5.













Discussion:



On the second point, I find myself totally agreeing with the theory behind coaching to lose. It does make a lot of sense to me and I see many parallels to this same approach in both large private businesses and government organizations.



On the first point, I find it very hard to agree to the theory behind pulling goalies earlier. Remember, these are extremely intelligent people behind these calculations and there are no reasons to believe that their probability estimates are incorrect. After all, all of their calculations are not theoretical - they are based on the actual NHL production numbers - all of the numbers, in fact. I do think there is something shaky and missing though. It could be that the players would take more shots on empty net resulting in more goals. But they have not so far in the time they have...



Anyway, here is the link. The letter is not very long. Enjoy!

The fact that no goalie has ever been pulled with 11 minutes to go and few if any at 5:40 to go suggests this is ALL theoretical.

[CaptainYooh]

Originally Posted by edslunch

The fact that no goalie has ever been pulled with 11 minutes to go and few if any at 5:40 to go suggests this is ALL theoretical.

No, that's the point of the article. It is not theoretical (empirical), but verified by actual data. I can't believe that the coaches don't know these probabilities already. Yet, they don't do it.

[DeluxeMoustache]

I haven’t read all of the other papers they cite.

Here is the problem. I am sure that the data amassed is based on a great number of situations where there are less than 30, 60, or 90 seconds on the clock. And it is being extrapolated.

When a goal is scored with the goalie pulled, how long does it usually take?
What is the projected frequency for change in possession when it is 6 on 5?
When the other team gets possession, what is their probability of scoring in the empty net?
For that matter, what is their strategy? Get a whistle or try to transition to offense? And how does that affect the short duration /late pull data.

I would expect that the only way to answer these questions would be for someone to start pulling the goalie at 5:40 to get a significant data set. And I suspect the likelihood of the team eventually losing possession and thus surrendering a goal when you pull with several minutes left may well outweigh the likelihood of surrendering a goal when limited time is left.

[DeluxeMoustache]

Or another way to say it - perhaps there is a model to be generated that suggests there is a window within the time frame of pulling the goalie to score, ideally, after which the odds of the other team gaining possession and the corresponding odds of being scored on increase to the point, that the advantage of pulling so early has dissipated.

And the value of the model, based on extrapolation of available data, is nullified.

So, for example, let’s say a team has 15 percent chance of scoring 6 on 5.
Let’s say the other team has
- 10 percent chance of gaining possession within 30 seconds
- 50 percent chance of gaining possession within 60 seconds
- 90 percent chance of gaining possession within 90 seconds

(I don’t know what these actual numbers are but somebody should be able to figure them out)

The point is that, as time elapses, the cumulative odds of the opponent gaining possession, and thus scoring, continue to increase.

Say they have a 30 percent chance of scoring if they get possession (?). Then the window is basically equal at around 60 seconds, and certainly something less than 90 seconds.

And if you are down by only one goal, in my mind you don’t pull the goalie with 5:40 left because the odds are the same within your window, then get progressively worse.

Then the logic actually goes the other way. If it is a 1 goal game and your window is 90 seconds, maybe you are best to wait until less than that 90 seconds.

[Regorium]

It's like how there has been multiple calculations that you should NEVER punt in football, yet somehow people pay at least one person to specifically execute a sub-optimal decision multiple times a game.

Math is weird when it comes to sports. I have no doubt that pulling the goalie early is a correct choice and teams usually wait too long to pull the goalie, but players aren't robots. Having a 5-0 game go to 10-0 because you're trying to play 6 on 5 has implications on the team dynamic past that single game.

[Weitz]

Coaches/GM's/any hockey people operate in the best way they think to keep their jobs, not to win IMO. Other sports suffer the same fate.

Which is why I think its stupid when the GM's get together to discuss rules changes. Why would they change rules to potentially threaten their job security?

[delayedreflex]

Originally Posted by DeluxeMoustache

I haven’t read all of the other papers they cite.

Here is the problem. I am sure that the data amassed is based on a great number of situations where there are less than 30, 60, or 90 seconds on the clock. And it is being extrapolated.

When a goal is scored with the goalie pulled, how long does it usually take?
What is the projected frequency for change in possession when it is 6 on 5?
When the other team gets possession, what is their probability of scoring in the empty net?
For that matter, what is their strategy? Get a whistle or try to transition to offense? And how does that affect the short duration /late pull data.

I would expect that the only way to answer these questions would be for someone to start pulling the goalie at 5:40 to get a significant data set. And I suspect the likelihood of the team eventually losing possession and thus surrendering a goal when you pull with several minutes left may well outweigh the likelihood of surrendering a goal when limited time is left.

Agreed - the paper extrapolates existing data, where the bulk of the dataset likely comes from 6 on 5 play in the last 2 minutes of the game. The paper assumes 1.18% chance for the team with the pulled goalie to score per 10 seconds, vs 2.58% chance for the team who is up by a goal. However, the team who is up by 1 is likely playing more conservatively in the last 2 minutes, simply trying not to concede a goal - so likely, if there was more time on the clock, the % of scoring would go up (both by the team with a pulled goalie, and by the team who is up). By the paper's own acknowledgement, the higher the scoring probability, the shorter the optimal window. So I don't know if the data can be extrapolated like the authors are doing in this paper.

Still, the general idea might be sound (coaches don't pull goalies as early as they should) even if the specific numbers are off. Pulling your goalie earlier will almost definitely hurt your team's goal differential, which is going to make your team look worse - but if you win one more game per year by pulling your goalie earlier, while losing three more games by 2 goals instead of 1, it is technically worth it. Patrick Roy did seem to see some success with earlier goalie pulls in his tenure, but I doubt it was a big enough sample size for us to see whether the strategy compared favourably with more conventional wisdom.

[corporatejay]

Originally Posted by delayedreflex

Agreed - the paper extrapolates existing data, where the bulk of the dataset likely comes from 6 on 5 play in the last 2 minutes of the game. The paper assumes 1.18% chance for the team with the pulled goalie to score per 10 seconds, vs 2.58% chance for the team who is up by a goal. However, the team who is up by 1 is likely playing more conservatively in the last 2 minutes, simply trying not to concede a goal - so likely, if there was more time on the clock, the % of scoring would go up (both by the team with a pulled goalie, and by the team who is up). By the paper's own acknowledgement, the higher the scoring probability, the shorter the optimal window. So I don't know if the data can be extrapolated like the authors are doing in this paper.

Still, the general idea might be sound (coaches don't pull goalies as early as they should) even if the specific numbers are off. Pulling your goalie earlier will almost definitely hurt your team's goal differential, which is going to make your team look worse - but if you win one more game per year by pulling your goalie earlier, while losing three more games by 2 goals instead of 1, it is technically worth it. Patrick Roy did seem to see some success with earlier goalie pulls in his tenure, but I doubt it was a big enough sample size for us to see whether the strategy compared favourably with more conventional wisdom.

I always found it weird that teams don't pull their goalie when they have a PP with less than 10 minutes left. That's likely your best chance to score.

[Jason14h]

Didn't Roy pulled his goalie with ~4 minutes left a few seasons ago? And then Hartley started pulling his goalie earlier as well?

Its seems that teams are willing to pull earlier now, but probably it should even swing more.

Just as football is seeing more teams go for it as the math suggests. It just takes a very long time to convince people the norm is incorrect.

Look how long it took teams in baseball to implement the shift, and now people complain it needs to be banned !

[Strange Brew]

Originally Posted by DeluxeMoustache

Or another way to say it - perhaps there is a model to be generated that suggests there is a window within the time frame of pulling the goalie to score, ideally, after which the odds of the other team gaining possession and the corresponding odds of being scored on increase to the point, that the advantage of pulling so early has dissipated.

And the value of the model, based on extrapolation of available data, is nullified.

So, for example, let’s say a team has 15 percent chance of scoring 6 on 5.
Let’s say the other team has
- 10 percent chance of gaining possession within 30 seconds
- 50 percent chance of gaining possession within 60 seconds
- 90 percent chance of gaining possession within 90 seconds

(I don’t know what these actual numbers are but somebody should be able to figure them out)

The point is that, as time elapses, the cumulative odds of the opponent gaining possession, and thus scoring, continue to increase.

Say they have a 30 percent chance of scoring if they get possession (?). Then the window is basically equal at around 60 seconds, and certainly something less than 90 seconds.

And if you are down by only one goal, in my mind you don’t pull the goalie with 5:40 left because the odds are the same within your window, then get progressively worse.

Then the logic actually goes the other way. If it is a 1 goal game and your window is 90 seconds, maybe you are best to wait until less than that 90 seconds.

I didn't look at the raw analysis but I would think they looked at this. No doubt the odds are stacked against you with no goalie, but at that point you're gambling for a different result than the current score. Presumably in coming up with 5:40 left they are incorporating the greater likelihood of getting scored against coupled with the greater likelihood of scoring over the entire time.

Although one thing they may not have considered. Pulling your goalie with 90 seconds left and you can essentially leave your best players on the ice. Not so with 5 minutes left. Of course same goes for the other guys. But the math changes.

[CaptainYooh]

Originally Posted by Strange Brew

...Pulling your goalie with 90 seconds left and you can essentially leave your best players on the ice. Not so with 5 minutes left. Of course same goes for the other guys. But the math changes.

I kinda liked this argument at first. But then, this would equally apply to both teams. Best attack vs best defense is only possible for so long. Things will even out, so the math shouldn't change much.

[united]

Like the Fourth Down Bot that has become somewhat popular in NFL circles, and is used (or closely replicated) by the Philadelphia Eagles, there is a Goalie Pull Bot for NHL games:

https://twitter.com/thepullbot?lang=en

https://twitter.com/user/status/981031419362402305

https://twitter.com/user/status/981034791918362624

The particular model is based on the first one the authors in OP talk about in the Appendix of their study, which accounts for home/away, estimated probability of getting a powerplay, and a small adjustment for estimated team strength.
http://moneypuck.com/about.htm#bot

[Passe La Puck]

So is this written by someone at one of the unsuccessful copycats? The assumption that every coach in the NHL is doing it so wrong just to save his job but some random finance guy figured it out is exceedingly unlikely and unbelievably arrogant.

[GGG]

I think the Hartley 3rd period comebacks were driven partly by early goalie pulls.

The big variable is it assumes that pulling the goalie at 5:40 to go has the same probability of scoring per minutes as with 90 seconds to go. This is not a valid assumption as strategies will change as people start pulling goalies early and you end up with a third line shift pulling that early instead of just 1/2nd line shifts.

That said I think the statement that goalies are pulled to late is valid in the same way that teams don't go for it enough on fourth down.

[Pointman]

Originally Posted by Passe La Puck

So is this written by someone at one of the unsuccessful copycats? The assumption that every coach in the NHL is doing it so wrong just to save his job but some random finance guy figured it out is exceedingly unlikely and unbelievably arrogant.

The assumption was that every coach is doing it so wrong not because they are dummies, but because their job depends on looking like a good coach rather than doing something risky and unconventional.

[Oil Stain]

Most NHL teams have a 10-20% chance to win the game when trailing after 2 periods. I wonder what percentage of these third period comebacks overcame more than a one goal deficit?

How many did the Flames win coming back 2 or more in the third? Or any team?

You look at the top teams winning percentage when leading after two periods.

http://www.nhl.com/stats/team?report...tgAfterTrail2p


Winnipeg as an example is 39 wins, 1 loss, and 1 OT loss.

If you are down by 2 with 10 minutes left in the third against the Jets you are essentially doomed regardless of what you do.

Pulling the goalie gives you a better chance to create offence. I mean you probably give up a goal the vast majority of the time with ten minutes of empty net, but if you come back to win like 3% more often its probably worth it from a statistical standpoint.

I guess the big downside is that when you give up a goal with an empty net it really takes all of the tension out of the game. Even if you are down by two, fans believe there is still a shot at a comeback.

An empty net goal against splashes cold water in their faces.

So I guess there is the question of whether making comebacks slightly more likely makes up for essentially ruining all the remaining entertainment value for a much larger percentage of games.

Having the goalie pulled with a minute left can make for some extremely exciting hockey. The race against the clock really adds to the excitement. Many times the defending team comes up with the puck and scores with like 20 seconds left. Alright games over anyway. Who cares?

Who though wants to sit and watch 4 minutes of garbage time hockey after a coach pulls his goalie at 5 minutes and is promptly scored on. Nobody.

Even a big offensive push at one goal down isn't going to put you on the edge of your seat because the attacking team still has 4-5 more minutes of potential opportunities.

Even if it's a slightly better winning tactic, it doesn't have anywhere near the same amount of fun/adrenaline factor IMO.

[mrkajz44]

Originally Posted by Regorium

It's like how there has been multiple calculations that you should NEVER punt in football, yet somehow people pay at least one person to specifically execute a sub-optimal decision multiple times a game.

Math is weird when it comes to sports. I have no doubt that pulling the goalie early is a correct choice and teams usually wait too long to pull the goalie, but players aren't robots. Having a 5-0 game go to 10-0 because you're trying to play 6 on 5 has implications on the team dynamic past that single game.

Here's a 538 story on the high school team that never punts:

http://fivethirtyeight.com/features/...o-never-punts/

Its a good watch and explains the numbers pretty well. I could see the same sort of thing with pulling the goalie type argument. Sometimes the "old way" is simply the wrong way, but no one has the balls to be the one person doing things differently.

[R0taryRocket]

At least the paper acknowledged the following. I would imagine GA would be horrible using this strategy.

A team down a goal with short time remaining gains a lot by scoring, and loses little if the other team scores as losing by two goals is no worse than losing by one

With the minor exception that the final tiebreaker for determining NHL playoff standings when two or more clubs are tied in points is the greatest differential between total goals for minus total goals against for the entire regular season. We don’t incorporate this into our model.

[Passe La Puck]

Originally Posted by Pointman

The assumption was that every coach is doing it so wrong not because they are dummies, but because their job depends on looking like a good coach rather than doing something risky and unconventional.

that is exactly what I said, and I find the idea offensive. These are the top coaches in the world, of course some are better than others and we question if some should even be in the league (GG), but the idea they they're all 100% making decisions focused on optics and not results is insane.

A far more reasonable explanation is that the model isn't very accurate. And that isn't surprising, these models take massive amounts of work on better framed problems (financial markets) and as the original blurb alluded to most models don't do great there either...