Bracketology Starts

[jkahn]

My big problem with playoffstatus is that their methodology isn't clear. How can I agree or disagree without knowing how they did it?

IIRC, CHN does a Monte Carlo analysis based on KRACH. I'm on board with that.

That is correct. I take the two KRACH values, and just use a random number generator to get the winner for that game, and every game. Then run it around 50,000 times, or as much as possible overnight.  But I do agree that KRACH might exaggerate things at the margins, given the relatively small sample sizes of past results we're dealing with.

I agree that using Adam's stated methodology is probably the best available way to look at it.  However, based upon last year, I do still believe there may be some flaws in the programming of the model.  Last year the model said we had a 98% chance of making the NCAA's going into the Clarkson series.  We had a 35% chance of losing Game 1 to Clarkson.  When we lost Game 1, our NCAA odds dropped to 65% per the model.  While I realize that there were other results that Friday night, if we had a 35% chance of being at 65% after the game, there's no way our odds before the game should have been any higher than 88%, and certainly not 98%.  If 98% were truly correct, than the 35% chance of a loss should have only dropped us to 94% at an average (35% of 94 plus 65% of 100 = 98).
Adam - it might be worthwhile looking into how the model handles playoff rounds and that fact that some series may go two games while others go three.  Does it automatically generate the third game only when necessary?  And does it appropriately handle subsequent match-ups?

[jfeath17]

"The outcome of each game in each simulation is determined by random draw, with the probability of victory for each team set by their relative KRACH ratings. So, if the simulation set included a contest between team A with KRACH 300 and team B with KRACH 100, team A will win the game in very close to 75% of the simulations. I don't simulate ties or home ice advantage."

https://blog.collegehockeyranked.com/2018/01/12/first-2018-pwr-forecasts-available/

Seems to be pretty much the same methodology as CHN.

[jfeath17]

I may be wrong, but I spent some time playing with the conference standings and I believe Cornell can clinch a first round bye with a sweep this weekend.

[KenP]

"The outcome of each game in each simulation is determined by random draw, with the probability of victory for each team set by their relative KRACH ratings. So, if the simulation set included a contest between team A with KRACH 300 and team B with KRACH 100, team A will win the game in very close to 75% of the simulations. I don't simulate ties or home ice advantage." Do the KRACH rating update after each game?  Or do you take today's number and hold constant?

https://blog.collegehockeyranked.com/2018/01/12/first-2018-pwr-forecasts-available/

Seems to be pretty much the same methodology as CHN.

[Scersk '97]

I may be wrong, but I spent some time playing with the conference standings and I believe Cornell can clinch a first round bye with a sweep this weekend.

Unfortunately, yes, you may be wrong. You can get Harvard, Colgate, Clarkson, and Union all tied at 30 and us at 29.

Pretty darn close to clinching, though!

[BearLover]

I don't agree at all with any of this line of criticism.  Roughly in order:

1. Just because intuitions disagree with odds doesn't mean the odds are wrong or the methodology is flawed.

No one is arguing this. But sometimes the odds are so, for lack of a better term, at odds with what we perceive to be true that heavy skepticism is warranted.

2.  We operate with a host of perception biases and our monkey brains are notoriously terrible at assigning relative likelihoods and proportions because we are strongly influenced by anecdotal experience.

Even my monkey brain can see that of our four games against our most likely second round opponents, Quinnipiac and Princeton, three of those games could have gone either way. The first Q game was decided on a crazy fluke goal. We were outshot 29-20. In the second game against Quinnipiac, we again won by one goal and were outshot 28-20. The first Princeton game we were outshot 25-22 and won on a goal with under seven minutes left. I put a lot of stake in models, far more than I do in my or any "expert's" "analysis," but this particular model is flawed. And as jkahn pointed out, it yields mathematically inconsistent results in addition to failing the eye test.

3. A model will of course "assume things keep on going as they have been" because that's the best guess of what will happen.  As long as the methodologically-correct degree of uncertainty (error) is built into the model, it's doing its job right.

Right, and my point is that the degree of uncertainty is not methodologically correct.

Why is a 96% chance of making Lake Placid "obviously" absurd?  Sure, it seems high, but what it means is 19 times out of 20 team x with our profile would advance.  That's still once in 20 that it doesn't.  That may well be in accord with historical actuals.

A math major should check me on this, but I believe having a 96% chance of winning a best-of-three series requires having an 88% chance of winning a single game. Even if Cornell were to finish the regular season as the best team in the country, I have a very hard time believing it would win almost 9/10 games against a middle-of-the-pack team. But we don't even know where the Red will finish.

The problem with models isn't modeling per se -- modeling works really well or Armstrong wouldn't have hit the moon.  What matters is taking care in choosing metrics and getting the probabilities right.  Statistics and probability theory have been working on each of those tasks respectively for a hundred years and as uncomfortable as it may be they now do a much, much better job of predicting than the eye test or common sense.

I love models. This particular model just isn't very good.

[adamw]

My big problem with playoffstatus is that their methodology isn't clear. How can I agree or disagree without knowing how they did it?

IIRC, CHN does a Monte Carlo analysis based on KRACH. I'm on board with that.

That is correct. I take the two KRACH values, and just use a random number generator to get the winner for that game, and every game. Then run it around 50,000 times, or as much as possible overnight.  But I do agree that KRACH might exaggerate things at the margins, given the relatively small sample sizes of past results we're dealing with.

I agree that using Adam's stated methodology is probably the best available way to look at it.  However, based upon last year, I do still believe there may be some flaws in the programming of the model.  Last year the model said we had a 98% chance of making the NCAA's going into the Clarkson series.  We had a 35% chance of losing Game 1 to Clarkson.  When we lost Game 1, our NCAA odds dropped to 65% per the model.  While I realize that there were other results that Friday night, if we had a 35% chance of being at 65% after the game, there's no way our odds before the game should have been any higher than 88%, and certainly not 98%.  If 98% were truly correct, than the 35% chance of a loss should have only dropped us to 94% at an average (35% of 94 plus 65% of 100 = 98).
Adam - it might be worthwhile looking into how the model handles playoff rounds and that fact that some series may go two games while others go three.  Does it automatically generate the third game only when necessary?  And does it appropriately handle subsequent match-ups?

Actually, I believe based on last year's conversation about this very situation, I did find a small flaw in the algorithm, and fixed it. I might have even talked about it here. I think that brought Cornell's odds down from 98% to something more like 92 -- but I can't remember exactly. I'm sure we could find it.

[ugarte]

I prefer this site. Shows where in the pairwise each team will finish based versus how many of their remaining games they win.

https://collegehockeyranked.com/forecast/pwrbywins/

says something about our SOS that we can win out and still drop to third (before the conference tournaments start).

[KGR11]

I don't agree at all with any of this line of criticism.  Roughly in order:

1. Just because intuitions disagree with odds doesn't mean the odds are wrong or the methodology is flawed.

No one is arguing this. But sometimes the odds are so, for lack of a better term, at odds with what we perceive to be true that heavy skepticism is warranted.

2.  We operate with a host of perception biases and our monkey brains are notoriously terrible at assigning relative likelihoods and proportions because we are strongly influenced by anecdotal experience.

Even my monkey brain can see that of our four games against our most likely second round opponents, Quinnipiac and Princeton, three of those games could have gone either way. The first Q game was decided on a crazy fluke goal. We were outshot 29-20. In the second game against Quinnipiac, we again won by one goal and were outshot 28-20. The first Princeton game we were outshot 25-22 and won on a goal with under seven minutes left. I put a lot of stake in models, far more than I do in my or any "expert's" "analysis," but this particular model is flawed. And as jkahn pointed out, it yields mathematically inconsistent results in addition to failing the eye test.

3. A model will of course "assume things keep on going as they have been" because that's the best guess of what will happen.  As long as the methodologically-correct degree of uncertainty (error) is built into the model, it's doing its job right.

Right, and my point is that the degree of uncertainty is not methodologically correct.

Why is a 96% chance of making Lake Placid "obviously" absurd?  Sure, it seems high, but what it means is 19 times out of 20 team x with our profile would advance.  That's still once in 20 that it doesn't.  That may well be in accord with historical actuals.

A math major should check me on this, but I believe having a 96% chance of winning a best-of-three series requires having an 88% chance of winning a single game. Even if Cornell were to finish the regular season as the best team in the country, I have a very hard time believing it would win almost 9/10 games against a middle-of-the-pack team. But we don't even know where the Red will finish.

The problem with models isn't modeling per se -- modeling works really well or Armstrong wouldn't have hit the moon.  What matters is taking care in choosing metrics and getting the probabilities right.  Statistics and probability theory have been working on each of those tasks respectively for a hundred years and as uncomfortable as it may be they now do a much, much better job of predicting than the eye test or common sense.

I love models. This particular model just isn't very good.

I checked your math, 88% for a single game is correct. But if we're the #1 team in the ECAC, we wouldn't be playing against the middle of the pack; we'd be playing against the worst team left, so the highest team we'd play is #8. On average, we're probably playing #9.

It sounds like Bearlover would prefer a model that incorporated more context on the games played. I'm imagining is some kind of weighted Scoring margin where you weight each goal for based on how good the opponents defense is and each goal against based on how opponents offense is. Not sure how easy that is to create, though.

[nshapiro]

I prefer this site. Shows where in the pairwise each team will finish based versus how many of their remaining games they win.

https://collegehockeyranked.com/forecast/pwrbywins/

says something about our SOS that we can win out and still drop to third (before the conference tournaments start).

I take these rankings with a BIG grain of salt, especially when https://collegehockeyranked.com/forecast/pwrchart/cornell/endofseason/ says there is a 100% certainty that going 1-7 means we finished exactly 10th in the pairwise.

[Swampy]

I prefer this site. Shows where in the pairwise each team will finish based versus how many of their remaining games they win.

https://collegehockeyranked.com/forecast/pwrbywins/

says something about our SOS that we can win out and still drop to third (before the conference tournaments start).

I take these rankings with a BIG grain of salt, especially when https://collegehockeyranked.com/forecast/pwrchart/cornell/endofseason/ says there is a 100% certainty that going 1-7 means we finished exactly 10th in the pairwise.

If I'm reading the diagram correctly, it's showing us ranked between 9 & 11 in the pairwise.

But I have another comment about the graph. Unless it's possible to finish ranked 9.5, why isn't this a side-by-side or stacked bar graph?

[BearLover]

I checked your math, 88% for a single game is correct. But if we're the #1 team in the ECAC, we wouldn't be playing against the middle of the pack; we'd be playing against the worst team left, so the highest team we'd play is #8. On average, we're probably playing #9.

Sorry, I meant "middle of the pack" as in the PWR, not the ECAC, because I was talking about us being the best team in the country (which is about where the PWR has us now), not the best team in the ECAC.

It sounds like Bearlover would prefer a model that incorporated more context on the games played. I'm imagining is some kind of weighted Scoring margin where you weight each goal for based on how good the opponents defense is and each goal against based on how opponents offense is. Not sure how easy that is to create, though.

I'd love a model like that, though that's not my biggest gripe with this model. Rather, the clearest problem is that the model does not sufficiently account for the natural randomness of a hockey game. It apparently treats slight favorites as big favorites, big favorites as overwhelming favorites, etc. Others in this thread have argued that maybe those big favorites are big favorites, and that the #1 seed in the ECAC is overwhelmingly likely to beat a team that's 30th in the PWR, but I have never seen odds for a hockey game even come close to assigning one team an 88% chance of winning. Someone who gambles should verify, but I would wager there hasn't been an NHL game this season where the odds were more than 70% in one direction.

Re-linking to this article, which states: " the better NHL team can expect to win 57% of matches played against an opponent on neutral ice."

It is true that there is a greater disparity between NCAA teams than there is between NHL teams, but 88% against an average team is just nuts.

[KGR11]

I checked your math, 88% for a single game is correct. But if we're the #1 team in the ECAC, we wouldn't be playing against the middle of the pack; we'd be playing against the worst team left, so the highest team we'd play is #8. On average, we're probably playing #9.

Sorry, I meant "middle of the pack" as in the PWR, not the ECAC, because I was talking about us being the best team in the country (which is about where the PWR has us now), not the best team in the ECAC.

It sounds like Bearlover would prefer a model that incorporated more context on the games played. I'm imagining is some kind of weighted Scoring margin where you weight each goal for based on how good the opponents defense is and each goal against based on how opponents offense is. Not sure how easy that is to create, though.

I'd love a model like that, though that's not my biggest gripe with this model. Rather, the clearest problem is that the model does not sufficiently account for the natural randomness of a hockey game. It apparently treats slight favorites as big favorites, big favorites as overwhelming favorites, etc. Others in this thread have argued that maybe those big favorites are big favorites, and that the #1 seed in the ECAC is overwhelmingly likely to beat a team that's 30th in the PWR, but I have never seen odds for a hockey game even come close to assigning one team an 88% chance of winning. Someone who gambles should verify, but I would wager there hasn't been an NHL game this season where the odds were more than 70% in one direction.

Re-linking to this article, which states: " the better NHL team can expect to win 57% of matches played against an opponent on neutral ice."

It is true that there is a greater disparity between NCAA teams than there is between NHL teams, but 88% against an average team is just nuts.

I think the easiest methodology would be to analyze how KRACH's pseudo-win-loss records match up with what actually happens. I'm imagining a dampening factor or equation could make it a better predictor.

Does anyone know if this has been tried?

[Tom Lento]

I checked your math, 88% for a single game is correct. But if we're the #1 team in the ECAC, we wouldn't be playing against the middle of the pack; we'd be playing against the worst team left, so the highest team we'd play is #8. On average, we're probably playing #9.

Sorry, I meant "middle of the pack" as in the PWR, not the ECAC, because I was talking about us being the best team in the country (which is about where the PWR has us now), not the best team in the ECAC.

It sounds like Bearlover would prefer a model that incorporated more context on the games played. I'm imagining is some kind of weighted Scoring margin where you weight each goal for based on how good the opponents defense is and each goal against based on how opponents offense is. Not sure how easy that is to create, though.

I'd love a model like that, though that's not my biggest gripe with this model. Rather, the clearest problem is that the model does not sufficiently account for the natural randomness of a hockey game. It apparently treats slight favorites as big favorites, big favorites as overwhelming favorites, etc. Others in this thread have argued that maybe those big favorites are big favorites, and that the #1 seed in the ECAC is overwhelmingly likely to beat a team that's 30th in the PWR, but I have never seen odds for a hockey game even come close to assigning one team an 88% chance of winning. Someone who gambles should verify, but I would wager there hasn't been an NHL game this season where the odds were more than 70% in one direction.

Re-linking to this article, which states: " the better NHL team can expect to win 57% of matches played against an opponent on neutral ice."

It is true that there is a greater disparity between NCAA teams than there is between NHL teams, but 88% against an average team is just nuts.

I think the easiest methodology would be to analyze how KRACH's pseudo-win-loss records match up with what actually happens. I'm imagining a dampening factor or equation could make it a better predictor.

Does anyone know if this has been tried?

I don't know if this has been tried, but it's a way to address the real modeling flaw here (if you even feel like it's a flaw worth addressing - there's a valid case to be made that this simulated odds thing is simply an extension of KRACH and therefore using all of their assumptions is, in fact, appropriate and whatever "bad" predictions come out are mainly a matter of academic interest).

The basic issue here is the model is taking KRACH predictors, which have some error associated with them, and then taking them as perfectly correct. It basically strips out the variance around the inputs, which leads to these kind of ridiculously overconfident predictive values. My guess is the distributions around those predictions are artificially small as a result, which is how Cornell can get 96% to make Placid.

Adding in an adjustment for KRACH's previously observed error is one option, although it's risky. We don't really know if KRACH's predictions are systematically biased with respect to the empirical reality of college hockey (intuitively, I expect this is the case, but it'd take quite a bit of data to figure that out).

Another option is to take KRACH as-is and update the model with additional information, similar to the way 538 does their CARMElo rankings for the NBA. In 538's case they have a ton of data to work with so they use player level predictors. For college hockey you might be stuck modifying by close (or even strength) possession in previous matchups and using that to adjust the KRACH-generated odds in the simulation.

[Jim Hyla]

Bracketology: No new teams in or out, but matchups starting to get interesting

This week's brackets

Midwest Regional (Allentown):
16 Canisius vs. 1 Notre Dame
10 Omaha vs. 6 Ohio State

East Regional (Bridgeport):
14 Western Michigan vs. 4 Cornell
11 Providence vs. 5 Minnesota State

West Regional (Sioux Falls):
15 Boston College vs. 2 St. Cloud State
9 Minnesota vs. 8 North Dakota

Northeast Regional (Worcester):
13 Northeastern vs. 3 Denver
12 Minnesota Duluth vs. 7 Clarkson

Conference breakdowns

NCHC — 6
Big Ten — 3
Hockey East — 3
ECAC Hockey — 2
WCHA — 1
Atlantic Hockey – 1

This week's movement:

Out: None

In: None