2018 ECAC Permutations

[adamw]

I wonder, is there anything like KRACH in any competitive team sport that anyone here believes does a good job predicting outcomes of individual games?

SRS is used by Sports Reference and its family of sites - which I also work for. It's essentially KRACH but with score differential taken into account. I don't necessarily think that's better or worse. I've suggested they add something that gives less weight to the difference as it increases. It's being considered. Otherwise, I don't know.

[abmarks]

it would be interesting to run a KRACH computation for last year's full NHL regular season, for example, and then see if the numbers pass people's gut checks or not.

[nshapiro]

Assume the actual probability distribution of Cornell's 1970 team going undefeated has an expected value of 0.95. In other words, if the team could replay the season an infinite number of times, 95% of the time it would go undefeated.

Why would you assume such a high number?

To achieve an expected value for the season of .95, you would have to assume a probability of winning each game is over .998.

If we assume the probability of Cornell '70 winning any game is .95, then the probability of an undefeated season is .226, and (don't pillory me please) that feels better.

[KGR11]

I think this discussion is getting old too, but since some people keep saying those criticizing the model are doing so based on "feel," I just want to say that we really aren't. (a) jfeath17 already showed KRACH overstates the chances of higher-ranked teams winning an individual game. (b) When combining several artificially inflated individual probabilities together (Cornell's chances of winning the quarters, semis, and finals) to form one joint probability (Cornell winning the ECAC tournament), you end up with a very, very overly inflated likelihood (the 55% chance of Cornell winning the ECAC). (c) There are no betting odds for any NHL game that come close to the odds this model is assigning many games every weekend.

Just to be clear, jfeath17's analysis shows that KRACH overstates the winning percentage for higher ranked teams if the KRACH winning % is greater than 65%. Otherwise, KRACH slightly underestimates the favorite team. Because of Cornell's impressive record, this nuance would only impact games against teams as good or better than Clarkson. It'd be interesting to see if that holds true for future seasons.

Using jfeath17's work would imply that the "true" probability of a game's outcome is based on the historical results of games with a similar KRACH difference between opponents. This assumption cannot be correct because each team is different, but how incorrect it is is dependent on the variability.

The current probability matrix says "What happens if teams maintain their current winning percentage going forward (adjusted for strength of schedule)?"
Using jfeath17's methodology, it would says "What happens if teams perform as well as the historical average team with similar KRACH ratings?"

Both are great questions with meaningful answers, but for forecasting, I would trust a matrix with jfeath17's adjustment slightly more than the current probability matrix (with the BIG assumption that the variation in results isn't ridiculous).

[Swampy]

Assume the actual probability distribution of Cornell's 1970 team going undefeated has an expected value of 0.95. In other words, if the team could replay the season an infinite number of times, 95% of the time it would go undefeated.

Why would you assume such a high number?

To achieve an expected value for the season of .95, you would have to assume a probability of winning each game is over .998.

If we assume the probability of Cornell '70 winning any game is .95, then the probability of an undefeated season is .226, and (don't pillory me please) that feels better.

t
It was for expository, illustrative purposes only. Tha's why I said, "Assume ... ."

In general, when we talk about using any instrument for prediction, we're making very strong ceteris paribus assumptions. We're assuming or predicting that nothing that can affect the outcome  changes.

Ironically, this may be legitimate for sports if we use fewer years rather than more. The more years we use, the greater the possibility of relevant differences -- rules changes, differences in conditioning, emphasis on speed vs size, injuries, etc.

[Beeeej]

Ironically, this may be legitimate for sports if we use fewer years rather than more. The more years we use, the greater the possibility of relevant differences -- rules changes, differences in conditioning, emphasis on speed vs size, injuries, etc.

Over time, I would expect that effect to be somewhat tempered by other multi-year factors such as recruiting parity and a more even distribution of injuries.

[abmarks]

I think this discussion is getting old too, but since some people keep saying those criticizing the model are doing so based on "feel," I just want to say that we really aren't. (a) jfeath17 already showed KRACH overstates the chances of higher-ranked teams winning an individual game. (b) When combining several artificially inflated individual probabilities together (Cornell's chances of winning the quarters, semis, and finals) to form one joint probability (Cornell winning the ECAC tournament), you end up with a very, very overly inflated likelihood (the 55% chance of Cornell winning the ECAC). (c) There are no betting odds for any NHL game that come close to the odds this model is assigning many games every weekend.

Just to be clear, jfeath17's analysis shows that KRACH overstates the winning percentage for higher ranked teams if the KRACH winning % is greater than 65%. Otherwise, KRACH slightly underestimates the favorite team. Because of Cornell's impressive record, this nuance would only impact games against teams as good or better than Clarkson. It'd be interesting to see if that holds true for future seasons.

Using jfeath17's work would imply that the "true" probability of a game's outcome is based on the historical results of games with a similar KRACH difference between opponents. This assumption cannot be correct because each team is different, but how incorrect it is is dependent on the variability.

The current probability matrix says "What happens if teams maintain their current winning percentage going forward (adjusted for strength of schedule)?"
Using jfeath17's methodology, it would says "What happens if teams perform as well as the historical average team with similar KRACH ratings?"

Both are great questions with meaningful answers, but for forecasting, I would trust a matrix with jfeath17's adjustment slightly more than the current probability matrix (with the BIG assumption that the variation in results isn't ridiculous).

Well said. Some people are taking jfeath's work as gospel, when in fact all we got was a simple chart showing krach win% v actual for specific, arbitrary bands.  Noone double-checked the work, either.

[adamw]

it would be interesting to run a KRACH computation for last year's full NHL regular season, for example, and then see if the numbers pass people's gut checks or not.

This is honestly an unnecessary exercise. For past results, it's hard to improve on KRACH. The KRACH ratings, if you played the schedule that already happened, would come out to the actual results.  That's the whole point of KRACH's existence.

[adamw]

Just for youse guys - I was able to work up NHL KRACH - for sh**s and giggles.

Code Select Expand


           Rating   RRWP   W-L-T      Pct    Ratio     SOS
 1   TBL   189.7   .6555   40-19-6   .6615   1.955    97.0
 2   VEG   189.4   .6552   39-20-4   .6508   1.864   101.7
 3   NSH   177.5   .6401   35-18-10  .6349   1.739   102.0
 4   BOS   154.6   .6076   36-21-5   .6210   1.638    94.4
 5   WPG   147.0   .5953   35-24-4   .5873   1.423   103.3
 6   TOR   127.3   .5603   33-25-8   .5606   1.276    99.8
 7   MIN   126.9   .5595   33-26-5   .5547   1.246   101.9
 8   PIT   124.6   .5548   34-27-4   .5538   1.241   100.4
 9   DAL   122.6   .5509   32-26-6   .5469   1.207   101.6
10   LAK   119.2   .5438   34-29-2   .5385   1.167   102.2
11   COL   118.5   .5424   33-28-2   .5397   1.172   101.1
12   PHI   118.2   .5418   32-25-7   .5547   1.246    94.9
13   WSH   117.4   .5402   33-27-4   .5469   1.207    97.3
14   SJS   111.0   .5261   31-27-7   .5308   1.131    98.1
15   STL   110.7   .5254   32-30-3   .5154   1.063   104.1
16   CGY   109.1   .5218   30-28-7   .5154   1.063   102.6
17   ANA   104.9   .5121   27-26-11  .5078   1.032   101.7
18   NJD   100.8   .5020   29-27-8   .5156   1.065    94.7
19   FLA   100.6   .5016   28-28-5   .5000   1.000   100.6
20   CBJ    92.8   .4812   26-28-10  .4844    .939    98.7
21   NYI    80.5   .4458   26-33-5   .4453    .803   100.2
22   NYR    78.3   .4389   25-32-7   .4453    .803    97.5
23   CAR    77.8   .4375   25-33-6   .4375    .778   100.1
24   CHI    75.3   .4295   26-36-2   .4219    .730   103.2
25   EDM    70.6   .4135   24-36-4   .4062    .684   103.1
26   VAN    68.4   .4061   24-37-3   .3984    .662   103.3
27   MTL    65.7   .3963   22-34-7   .4048    .680    96.6
28   DET    62.1   .3828   22-36-5   .3889    .636    97.6
29   OTT    59.1   .3711   19-35-8   .3710    .590   100.2
30   ARI    52.4   .3431   18-39-6   .3333    .500   104.8
31   BUF    46.8   .3177   20-43-1   .3203    .471    99.2


The range is obviously much more narrow, which would make odds much lower even for top teams.

Compare to current NCAA range ... 533 to 15

[Trotsky]

That really quantifies how much Buffalo sucks.

[Trotsky]

it would be interesting to run a KRACH computation for last year's full NHL regular season, for example, and then see if the numbers pass people's gut checks or not.

This is honestly an unnecessary exercise. For past results, it's hard to improve on KRACH. The KRACH ratings, if you played the schedule that already happened, would come out to the actual results.  That's the whole point of KRACH's existence.

How about this exercise.  Number all the NC$$ games in chron order.  Calculate KRACH from the odd number games.  Now compare how the even numbered games turned out against KRACH "predictions."

I know, I know, I know that KRACH reviews a data set and is not designed to be predictive.  But... can you do that for shits and giggles?  I'm not even sure how we'd interpret the results.  What constitutes a reliable or unreliable percentage of accuracy?  I mean, hopefully it's over 50%.  Hopefully it's better than just taking whoever has the better winning percentage excluding games against each other.

Another method: start on game 1 and just march through the list constantly recalculating KRACH and using that as the prediction against the next game.  Or since obviously KRACH gets better as the season goes on, iterate through say the first 10% of games and only start predicting after that.  Now get your accuracy score.  That's truly abusing KRACH as predictive.  

[BearLover]

Is KRACH not empirically based?

The future win probabilities inferred from KRACH aren't, because they're not verifiable by observation/experience.

It is not certain that looking at things beyond wins and losses is any better. Goal differential has major flaws, and might not mean much. Shot differential has its own issues, but could be a decent factor. Honestly, I'm not all that interested in things like goal and shot differential.

I think in the hockey analytics world it actually is pretty certain that looking at things beyond wins and losses is better.

I think it's helpful to think of it this way: only looking at game outcomes is a very small sample size. Goals, of which there are several in the average game, is a bigger sample. Shots is the biggest sample of all. And, in fact, shot differential, in serving as the best proxy we have for possession, does a tremendous job of measuring the strength of a team and thereby predicting the outcome of a hockey game.

This discussion often comes up on here when Cornell has a better record than its possession numbers would suggest and half of ELynah thinks the team will regress and the other half thinks they won't. Which necessarily leads to a discussion of whether possession is the be-all-end-all in college like it is in the pros. I won't rehash the arguments here, but shot differential is still so heavily correlated with wins in college that I highly doubt there exists a more predictive stat over the course of a regular season than shot differential.

Well said. Some people are taking jfeath's work as gospel, when in fact all we got was a simple chart showing krach win% v actual for specific, arbitrary bands. Noone double-checked the work, either.

jfeath's work is just a very preliminary exercise that confirms what many of us suspected when we looked at some of the numbers these prediction models were pumping out. It also comports with NHL betting odds. NHL betting odds almost never give a team a less than a 1-in-3 chance of winning. Yet a few weeks ago the KRACH-based model was giving Cornell an 80% chance of beating Union and Harvard!

[KGR11]

it would be interesting to run a KRACH computation for last year's full NHL regular season, for example, and then see if the numbers pass people's gut checks or not.

This is honestly an unnecessary exercise. For past results, it's hard to improve on KRACH. The KRACH ratings, if you played the schedule that already happened, would come out to the actual results.  That's the whole point of KRACH's existence.

How about this exercise.  Number all the NC$$ games in chron order.  Calculate KRACH from the odd number games.  Now compare how the even numbered games turned out against KRACH "predictions."

I know, I know, I know that KRACH reviews a data set and is not designed to be predictive.  But... can you do that for shits and giggles?  I'm not even sure how we'd interpret the results.  What constitutes a reliable or unreliable percentage of accuracy?  I mean, hopefully it's over 50%.  Hopefully it's better than just taking whoever has the better winning percentage excluding games against each other.

Another method: start on game 1 and just march through the list constantly recalculating KRACH and using that as the prediction against the next game.  Or since obviously KRACH gets better as the season goes on, iterate through say the first 10% of games and only start predicting after that.  Now get your accuracy score.  That's truly abusing KRACH as predictive.  

I think your second method is essentially what jfeath did, right?

[billhoward]

Sometimes I think it's pronounced krock.

[adamw]

Is KRACH not empirically based?

The future win probabilities inferred from KRACH aren't, because they're not verifiable by observation/experience.

What future probabilities of any kind are verifiable?

It is not certain that looking at things beyond wins and losses is any better. Goal differential has major flaws, and might not mean much. Shot differential has its own issues, but could be a decent factor. Honestly, I'm not all that interested in things like goal and shot differential.

I think in the hockey analytics world it actually is pretty certain that looking at things beyond wins and losses is better.

There is no need to quote me articles about analytics. I deal with NHL analytics all day for my "real" job. The analytics community has also, finally, thank goodness, moved beyond its original rudimentary hypotheses about how hockey works. Shot differential as a proxy for possession was a nice tool in the toolbelt, but had/has a long way to go to create real understanding. There is shot quality, location data, rolling score effects, etc... finally being taken into consideration, and of course there is a lot in hockey that simply can't be measured yet.  So while shot differential displayed some correlation to better predicting wins/losses than past wins and losses, it's really very rudimentary and there's plenty more to do.

However, you glossed over the fact that I said "goal differential" first. I don't know of any model that takes into account shot differential in ranking systems.  Goal differential is another thing.  There have been plenty of them that do.  And that debate has gone on forever.  My point was that goal differential has numerous flaws when it comes to hockey team ratings, which is why it's perfectly valid to ignore it when it comes to ratings systems, and probably predictive models.  I also clearly said that shot differential could be a "decent factor" but has issues. So I'm not sure why the need to inform me that looking beyond wins and losses may be better. Well aware. No one ever said otherwise.

I think it's helpful to think of it this way: only looking at game outcomes is a very small sample size. Goals, of which there are several in the average game, is a bigger sample. Shots is the biggest sample of all. And, in fact, shot differential, in serving as the best proxy we have for possession, does a tremendous job of measuring the strength of a team and thereby predicting the outcome of a hockey game.

Please stop with the Analytic-splaining ... Believe me, we all understand about sample sizes. Again, goal differential in hockey is flawed. The greater sample size there is not necessarily an improvement. And I would not call shot differential metrics doing a "tremendous job" ... It does a better job. Not a tremendous job. There are more factors.  But sure, on the team level, it holds some weight.  If it can be incorporated into predictive models, then great. But it's not a panacea.

It also comports with NHL betting odds. NHL betting odds almost never give a team a less than a 1-in-3 chance of winning. Yet a few weeks ago the KRACH-based model was giving Cornell an 80% chance of beating Union and Harvard!

There's nothing more flawed than quoting betting odds, which bear no resemblance to anything except where money goes. I'm pretty sure the NHL KRACH figures I posted earlier demonstrate that the variance between NHL teams is far smaller than college teams. So comparing betting odds of NHL games to college possibilities is silly. Of course betting odds are never that wide on NHL games.

Again - we all get that there are better ways to do things. But I don't understand all the bellyaching about it. Many of your solutions have plenty of issues themselves.  Come up with a model, and lay out the math, have it reviewed for problems, and I'll be more than happy to put it together. I haven't seen anyone be willing to do that yet.