2018 ECAC Permutations
Posted by Give My Regards
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: Trotsky (---.dc.dc.cox.net)
Date: March 02, 2018 05:32PM
adamw
abmarks
it would be interesting to run a KRACH computation for last year's full NHL regular season, for example, and then see if the numbers pass people's gut checks or not.
This is honestly an unnecessary exercise. For past results, it's hard to improve on KRACH. The KRACH ratings, if you played the schedule that already happened, would come out to the actual results. That's the whole point of KRACH's existence.
How about this exercise. Number all the NC$$ games in chron order. Calculate KRACH from the odd number games. Now compare how the even numbered games turned out against KRACH "predictions."
I know, I know, I know that KRACH reviews a data set and is not designed to be predictive. But... can you do that for shits and giggles? I'm not even sure how we'd interpret the results. What constitutes a reliable or unreliable percentage of accuracy? I mean, hopefully it's over 50%. Hopefully it's better than just taking whoever has the better winning percentage excluding games against each other.
Another method: start on game 1 and just march through the list constantly recalculating KRACH and using that as the prediction against the next game. Or since obviously KRACH gets better as the season goes on, iterate through say the first 10% of games and only start predicting after that. Now get your accuracy score. That's truly abusing KRACH as predictive.
Edited 5 time(s). Last edit at 03/02/2018 05:38PM by Trotsky.
Re: 2018 ECAC Permutations
Posted by: BearLover (---.nycmny.ftas.verizon.net)
Date: March 03, 2018 01:04AM
The future win probabilities inferred from KRACH aren't, because they're not verifiable by observation/experience.adamw
Is KRACH not empirically based?
I think in the hockey analytics world it actually is pretty certain that looking at things beyond wins and losses is better.adamw
It is not certain that looking at things beyond wins and losses is any better. Goal differential has major flaws, and might not mean much. Shot differential has its own issues, but could be a decent factor. Honestly, I'm not all that interested in things like goal and shot differential.
I think it's helpful to think of it this way: only looking at game outcomes is a very small sample size. Goals, of which there are several in the average game, is a bigger sample. Shots is the biggest sample of all. And, in fact, shot differential, in serving as the best proxy we have for possession, does a tremendous job of measuring the strength of a team and thereby predicting the outcome of a hockey game.
This discussion often comes up on here when Cornell has a better record than its possession numbers would suggest and half of ELynah thinks the team will regress and the other half thinks they won't. Which necessarily leads to a discussion of whether possession is the be-all-end-all in college like it is in the pros. I won't rehash the arguments here, but shot differential is still so heavily correlated with wins in college that I highly doubt there exists a more predictive stat over the course of a regular season than shot differential.
jfeath's work is just a very preliminary exercise that confirms what many of us suspected when we looked at some of the numbers these prediction models were pumping out. It also comports with NHL betting odds. NHL betting odds almost never give a team a less than a 1-in-3 chance of winning. Yet a few weeks ago the KRACH-based model was giving Cornell an 80% chance of beating Union and Harvard!abmarks
Well said. Some people are taking jfeath's work as gospel, when in fact all we got was a simple chart showing krach win% v actual for specific, arbitrary bands. Noone double-checked the work, either.
Edited 1 time(s). Last edit at 03/03/2018 01:07AM by BearLover.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: KGR11 (---.dyn.optonline.net)
Date: March 03, 2018 08:07AM
Trotsky
adamw
abmarks
it would be interesting to run a KRACH computation for last year's full NHL regular season, for example, and then see if the numbers pass people's gut checks or not.
This is honestly an unnecessary exercise. For past results, it's hard to improve on KRACH. The KRACH ratings, if you played the schedule that already happened, would come out to the actual results. That's the whole point of KRACH's existence.
How about this exercise. Number all the NC$$ games in chron order. Calculate KRACH from the odd number games. Now compare how the even numbered games turned out against KRACH "predictions."
I know, I know, I know that KRACH reviews a data set and is not designed to be predictive. But... can you do that for shits and giggles? I'm not even sure how we'd interpret the results. What constitutes a reliable or unreliable percentage of accuracy? I mean, hopefully it's over 50%. Hopefully it's better than just taking whoever has the better winning percentage excluding games against each other.
Another method: start on game 1 and just march through the list constantly recalculating KRACH and using that as the prediction against the next game. Or since obviously KRACH gets better as the season goes on, iterate through say the first 10% of games and only start predicting after that. Now get your accuracy score. That's truly abusing KRACH as predictive.
I think your second method is essentially what jfeath did, right?
Re: 2018 ECAC Permutations
Posted by: billhoward (---.nwrk.east.verizon.net)
Date: March 03, 2018 12:12PM
Sometimes I think it's pronounced krock.
Re: 2018 ECAC Permutations
Posted by: adamw (---.phlapa.fios.verizon.net)
Date: March 03, 2018 12:35PM
BearLover
The future win probabilities inferred from KRACH aren't, because they're not verifiable by observation/experience.adamw
Is KRACH not empirically based?
What future probabilities of any kind are verifiable?
BearLover
I think in the hockey analytics world it actually is pretty certain that looking at things beyond wins and losses is better.adamw
It is not certain that looking at things beyond wins and losses is any better. Goal differential has major flaws, and might not mean much. Shot differential has its own issues, but could be a decent factor. Honestly, I'm not all that interested in things like goal and shot differential.
There is no need to quote me articles about analytics. I deal with NHL analytics all day for my "real" job. The analytics community has also, finally, thank goodness, moved beyond its original rudimentary hypotheses about how hockey works. Shot differential as a proxy for possession was a nice tool in the toolbelt, but had/has a long way to go to create real understanding. There is shot quality, location data, rolling score effects, etc... finally being taken into consideration, and of course there is a lot in hockey that simply can't be measured yet. So while shot differential displayed some correlation to better predicting wins/losses than past wins and losses, it's really very rudimentary and there's plenty more to do.
However, you glossed over the fact that I said "goal differential" first. I don't know of any model that takes into account shot differential in ranking systems. Goal differential is another thing. There have been plenty of them that do. And that debate has gone on forever. My point was that goal differential has numerous flaws when it comes to hockey team ratings, which is why it's perfectly valid to ignore it when it comes to ratings systems, and probably predictive models. I also clearly said that shot differential could be a "decent factor" but has issues. So I'm not sure why the need to inform me that looking beyond wins and losses may be better. Well aware. No one ever said otherwise.
BearLover
I think it's helpful to think of it this way: only looking at game outcomes is a very small sample size. Goals, of which there are several in the average game, is a bigger sample. Shots is the biggest sample of all. And, in fact, shot differential, in serving as the best proxy we have for possession, does a tremendous job of measuring the strength of a team and thereby predicting the outcome of a hockey game.
Please stop with the Analytic-splaining ... Believe me, we all understand about sample sizes. Again, goal differential in hockey is flawed. The greater sample size there is not necessarily an improvement. And I would not call shot differential metrics doing a "tremendous job" ... It does a better job. Not a tremendous job. There are more factors. But sure, on the team level, it holds some weight. If it can be incorporated into predictive models, then great. But it's not a panacea.
BearLover
It also comports with NHL betting odds. NHL betting odds almost never give a team a less than a 1-in-3 chance of winning. Yet a few weeks ago the KRACH-based model was giving Cornell an 80% chance of beating Union and Harvard!
There's nothing more flawed than quoting betting odds, which bear no resemblance to anything except where money goes. I'm pretty sure the NHL KRACH figures I posted earlier demonstrate that the variance between NHL teams is far smaller than college teams. So comparing betting odds of NHL games to college possibilities is silly. Of course betting odds are never that wide on NHL games.
Again - we all get that there are better ways to do things. But I don't understand all the bellyaching about it. Many of your solutions have plenty of issues themselves. Come up with a model, and lay out the math, have it reviewed for problems, and I'll be more than happy to put it together. I haven't seen anyone be willing to do that yet.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: BearLover (---.nycmny.ftas.verizon.net)
Date: March 03, 2018 02:26PM
What I mean to say is that the probabilities derived purely from KRACH are not back-tested against actual results, in the way 538's and a model like this one's are. The KRACH-based model instead takes what has happened in the past and extrapolates it into the future, and no one even checks to see how much it misses by.
What future probabilities of any kind are verifiable?
I agree with all of this. But just because shots as a statistic is rudimentary doesn't mean it isn't the best tool we have right now.adamw
There is no need to quote me articles about analytics. I deal with NHL analytics all day for my "real" job. The analytics community has also, finally, thank goodness, moved beyond its original rudimentary hypotheses about how hockey works. Shot differential as a proxy for possession was a nice tool in the toolbelt, but had/has a long way to go to create real understanding. There is shot quality, location data, rolling score effects, etc... finally being taken into consideration, and of course there is a lot in hockey that simply can't be measured yet. So while shot differential displayed some correlation to better predicting wins/losses than past wins and losses, it's really very rudimentary and there's plenty more to do.
Again, you're conflating "is rudimentary/has its own sets of issues" with "is worse." Goals and shots aren't perfect, but they're better predictors than wins.adamw
However, you glossed over the fact that I said "goal differential" first. I don't know of any model that takes into account shot differential in ranking systems. Goal differential is another thing. There have been plenty of them that do. And that debate has gone on forever. My point was that goal differential has numerous flaws when it comes to hockey team ratings, which is why it's perfectly valid to ignore it when it comes to ratings systems, and probably predictive models. I also clearly said that shot differential could be a "decent factor" but has issues. So I'm not sure why the need to inform me that looking beyond wins and losses may be better. Well aware. No one ever said otherwise.
Sorry about the analytic-splaining, but I can't recall an analytics study/article in the past six or so years I've been following this that concluded wins/losses is a better metric of future success than goals, and I don't know if I even recall an article/study that concluded goals were a better metric than shots. In fact, it seems every article/study leads with the assumption that Corsi/Fenwick is the best predictor we currently have, and goes from there. So you writing that you are not interested in a model that looks at goals/shots rather than wins suggested to me you aren't as familiar with current prediction models. I was wrong about your lack of familiarity, so you are welcome to post things that would back up your disdain for shot differential as a predictive stat relative to win% and goal-differential.adamw
Please stop with the Analytic-splaining ...
I posted betting odds because I wasn't aware of an actual NHL prediction model that gave probabilities for individual games. I've since found one, and it turns out I was wrong about the upper bounds of hockey probabilities: while the majority of its predictions are closer to 50% than those of KRACH, this model yields probabilities for certain games that are as high as 80%. The model seems to care less about a team's entire body of work and instead weights factors such as recent performance and starting goalie quality very highly--though it doesn't appear to release its entire methodology. The model claims to be about 60% accurate, which is about as good as it gets for NHL prediction models. On the other hand, the fact that this model is deriving these relatively lopsided probabilities not from total record but from all these other stats doesn't really help the case for a KRACH model, which looks only at past win% (and in fact there are a lot of probabilities from this model that show a matchup between teams with similar records as lopsided).adamw
There's nothing more flawed than quoting betting odds, which bear no resemblance to anything except where money goes.
Yes, this is true. But is the gap between Cornell and Union/Harvard as big as the largest gap between any two NHL teams?adamw
I'm pretty sure the NHL KRACH figures I posted earlier demonstrate that the variance between NHL teams is far smaller than college teams.
Edited 1 time(s). Last edit at 03/03/2018 02:27PM by BearLover.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: jfeath17 (---.bstnma.fios.verizon.net)
Date: March 03, 2018 03:54PM
KGR11
Trotsky
Another method: start on game 1 and just march through the list constantly recalculating KRACH and using that as the prediction against the next game. Or since obviously KRACH gets better as the season goes on, iterate through say the first 10% of games and only start predicting after that. Now get your accuracy score. That's truly abusing KRACH as predictive.
I think your second method is essentially what jfeath did, right?
Yes, that is basically what I did except I stepped through week by week and started in January. I now have updated data which includes 4 seasons and steps through on a daily basis starting in January.
BearLover
The future win probabilities inferred from KRACH aren't, because they're not verifiable by observation/experience.adamw
Is KRACH not empirically based?
What future probabilities of any kind are verifiable?
While you can't perfectly verify a prediction model, you can get an idea of its performance by separating the past data into training and testing sets. The fact that we are trying to predict probabilities and not simple classification does make it much more difficult to evaluate the performance. For classification problems the predictor is either right or wrong so it is easy to state a accuracy percentage. We however cannot directly observe the outcome probability of some matchup but only outcome of one trial of this matchup. This brings us to what I am attempting to do. By looking at the outcomes of many games with a similar krach predicted winning percentage we can come up with an estimate for the actual winning percentage of a team in this matchup.
My methodology for this was to use a gaussian weighted average of the games centered at varying krach probabilities. I calculated the winning percentage using these weights. I also used the weights to come up with average krach probability (this doesn't necessarily line up with the center of the gaussian particularly at the endpoints where all the games are to one side or the other). This is basically the logical extrapolation of the binning that was suggested earlier in the thread. The binning was actually the first analysis I did but the data didn't look good. I think the major improvement here is not that I am using the gaussian to come up with weights (that is probably overkill), but that I am using the weighted average of the krach probabilities rather than the center of the bin. What this trend line looks like can be changed significantly by changing the std dev of the gaussian (effectively changing the bin size). Basically the larger the bin the more underfit and the smaller the more overfit.
I also sought to measure the performance of KRACH in another way by looking at the R2 (Coefficient of Determination) The Wikipedia page is a pretty good explanation of this. The R^2 value can be looked at as the percentage of variance in the dependent variable (game outcomes) that can be explained by the independent variable (krach probability, etc..). These probabilities are all very low which makes sense since there is a lot of variability in the outcome of hockey games.
Independent Variable | R^2
--------------------------------------------------
Krach Probability | 0.023
Logistic Regression | 0.100
Linear Fit on Gaussian Average (0.1) | 0.112
(y=.749x+.126)
Another improvement I made was to include the inverse of each game (prob = 1-prob and swap wins and losses). This improves the fit around 0.5 since it is a little nonsensical if the matchup of two equal KRACH teams is not 0.5 in a model only dependent on KRACH.
One final point which I think has been established at this point, but I want to make sure we are all on the same page. Predictive models are going to have some subjectivity built into them. It is great that KRACH has no subjectivity and is a mathematically pure ranking for its goal of NCAA seeding since that needs to be "fair" and should be based on the actual outcomes of the games. However when creating a predictive model, we unfortunately do not have the luxury of a mathematically pure system. There are parameters and methods that must be chosen both when designing a predictor and measuring the performance. It is the designers goal to choose these such that the predictor is not over/underfit or have any bias's built in.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: adamw (---.phlapa.fios.verizon.net)
Date: March 03, 2018 07:06PM
BearLover
Again, you're conflating "is rudimentary/has its own sets of issues" with "is worse." Goals and shots aren't perfect, but they're better predictors than wins.
Actually, I think you're the only one conflating anything, because I never said "worse" - so I'm not sure where this is coming from.
This thread is very enjoyable to me, and I want to hear all of these things. And I'm pretty sure I've been clear that KRACH can be improved upon. And I agree with what you say, in general. What has rubbed me the wrong way with your posts is your unnecessary (to me) vehemence against KRACH, the twisting of what I've said, and the high-level of self-confidence in what you're saying. A little humility is helpful here because no one really knows.
BearLover
Sorry about the analytic-splaining, but I can't recall an analytics study/article in the past six or so years I've been following this that concluded wins/losses is a better metric of future success than goals, and I don't know if I even recall an article/study that concluded goals were a better metric than shots. In fact, it seems every article/study leads with the assumption that Corsi/Fenwick is the best predictor we currently have, and goes from there. So you writing that you are not interested in a model that looks at goals/shots rather than wins suggested to me you aren't as familiar with current prediction models. I was wrong about your lack of familiarity, so you are welcome to post things that would back up your disdain for shot differential as a predictive stat relative to win% and goal-differential.
The problem is you continuing to insist I said things that I never said. You are claiming that I am against shots/goals differential, when I never said anything of the sort. I just said it has its own issues. Particularly goal differential. Is a 6-2 win more indicative of quality than 6-4? Not necessarily in my opinion, and I think studies of such are flawed. Of course, there's empty-net goals to take into account, which is unique to hockey. Of course, those can be weeded out.
The older Analytics studies have not just been improved upon - in some cases, they have been downright contradicted. There is too much more involved in hockey than to be so confident in what these things are telling us so far. I've had issues with these studies from day one. And thankfully now they are being improved upon and we are finding out different things. Heck, Bill James spent most of his career telling everyone that clutch didn't exist, and now he spent the last year telling everyone that we should taken into account situational hitting when it comes to voting for the MVP Award.
So again, all I'm saying is, there is room for all of this in the discussion - but the insistence that you "know" "for a fact" that you are right with this, and the KRACH is just a load of garbage, is where it rubs me the wrong way.
BearLover
I posted betting odds because I wasn't aware of an actual NHL prediction model that gave probabilities for individual games. I've since found one, and it turns out I was wrong about the upper bounds of hockey probabilities
My site, hockey-reference.com, publishes Win Probabilities every day. As I mentioned upthread, it's based upon SRS, which is similar to KRACH, but takes into account goal differential.[/quote]
BearLover
Yes, this is true. But is the gap between Cornell and Union/Harvard as big as the largest gap between any two NHL teams?
Doubtful, but I don't think anyone here said it was. We also, however, don't know that, either way.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: adamw (---.phlapa.fios.verizon.net)
Date: March 03, 2018 07:12PM
jfeath17
One final point which I think has been established at this point, but I want to make sure we are all on the same page. Predictive models are going to have some subjectivity built into them. It is great that KRACH has no subjectivity and is a mathematically pure ranking for its goal of NCAA seeding since that needs to be "fair" and should be based on the actual outcomes of the games. However when creating a predictive model, we unfortunately do not have the luxury of a mathematically pure system. There are parameters and methods that must be chosen both when designing a predictor and measuring the performance. It is the designers goal to choose these such that the predictor is not over/underfit or have any bias's built in.
Absolutely. That is why I think it's a mistake to be so vehement about some better model, and so dismissive of what the current model is doing. There is something better, I'm sure, but exactly what is debatable.
BTW - I can't read your charts, so I have no idea what they're telling me. If there's an English translation, feel free.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: KGR11 (---.dyn.optonline.net)
Date: March 03, 2018 08:26PM
jfeath17
While you can't perfectly verify a prediction model, you can get an idea of its performance by separating the past data into training and testing sets. The fact that we are trying to predict probabilities and not simple classification does make it much more difficult to evaluate the performance. For classification problems the predictor is either right or wrong so it is easy to state a accuracy percentage. We however cannot directly observe the outcome probability of some matchup but only outcome of one trial of this matchup. This brings us to what I am attempting to do. By looking at the outcomes of many games with a similar krach predicted winning percentage we can come up with an estimate for the actual winning percentage of a team in this matchup.
My methodology for this was to use a gaussian weighted average of the games centered at varying krach probabilities. I calculated the winning percentage using these weights. I also used the weights to come up with average krach probability (this doesn't necessarily line up with the center of the gaussian particularly at the endpoints where all the games are to one side or the other). This is basically the logical extrapolation of the binning that was suggested earlier in the thread. The binning was actually the first analysis I did but the data didn't look good. I think the major improvement here is not that I am using the gaussian to come up with weights (that is probably overkill), but that I am using the weighted average of the krach probabilities rather than the center of the bin. What this trend line looks like can be changed significantly by changing the std dev of the gaussian (effectively changing the bin size). Basically the larger the bin the more underfit and the smaller the more overfit.
I also sought to measure the performance of KRACH in another way by looking at the R2 (Coefficient of Determination) The Wikipedia page is a pretty good explanation of this. The R^2 value can be looked at as the percentage of variance in the dependent variable (game outcomes) that can be explained by the independent variable (krach probability, etc..). These probabilities are all very low which makes sense since there is a lot of variability in the outcome of hockey games.
Independent Variable | R^2
--------------------------------------------------
Krach Probability | 0.023
Logistic Regression | 0.100
Linear Fit on Gaussian Average (0.1) | 0.112
(y=.749x+.126)
Another improvement I made was to include the inverse of each game (prob = 1-prob and swap wins and losses). This improves the fit around 0.5 since it is a little nonsensical if the matchup of two equal KRACH teams is not 0.5 in a model only dependent on KRACH.
One final point which I think has been established at this point, but I want to make sure we are all on the same page. Predictive models are going to have some subjectivity built into them. It is great that KRACH has no subjectivity and is a mathematically pure ranking for its goal of NCAA seeding since that needs to be "fair" and should be based on the actual outcomes of the games. However when creating a predictive model, we unfortunately do not have the luxury of a mathematically pure system. There are parameters and methods that must be chosen both when designing a predictor and measuring the performance. It is the designers goal to choose these such that the predictor is not over/underfit or have any bias's built in.
Many thanks to jfeath for this awesome work. One think I'd be curious to know is how the R-squared for the KRACH, Logistic Regression, and Linear Fit on Gaussian Average varies from year to year. This could give an indication of the extent (if any) that KRACH is a lesser predictor than your outputs.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: jfeath17 (---.bstnma.fios.verizon.net)
Date: March 03, 2018 08:35PM
adamw
BTW - I can't read your charts, so I have no idea what they're telling me. If there's an English translation, feel free.
My one sentence summary is that the current KRACH probabilities are biased towards the higher ranked team, a very simple modification which would greatly increase the accuracy is to change the formula to P(A Winning) = .749*(KRACH_A/(KRACH_A+KRACH_B)) + .126
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: jfeath17 (---.bstnma.fios.verizon.net)
Date: March 03, 2018 08:37PM
KGR11
Many thanks to jfeath for this awesome work. One think I'd be curious to know is how the R-squared for the KRACH, Logistic Regression, and Linear Fit on Gaussian Average varies from year to year. This could give an indication of the extent (if any) that KRACH is a lesser predictor than your outputs.
I can try this out to just as a sanity check to make sure the models aren't overfitting. I may even do it on a completely new season to be sure.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: adamw (---.phlapa.fios.verizon.net)
Date: March 03, 2018 08:38PM
jfeath17
adamw
BTW - I can't read your charts, so I have no idea what they're telling me. If there's an English translation, feel free.
My one sentence summary is that the current KRACH probabilities are biased towards the higher ranked team, a very simple modification which would greatly increase the accuracy is to change the formula to P(A Winning) = .749*(KRACH_A/(KRACH_A+KRACH_B)) + .126
Wait, that's English?
... If you want to work with me on something going forward, feel free to drop me a line. adamw@collegehockeynews.com (same for anyone else who has chimed in here with something concrete to offer)
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: jkahn (---.hsd1.il.comcast.net)
Date: March 03, 2018 10:28PM
Basically, what's being suggested here is to use KRACH for 75% of the probability and split the other 25% equally. I was actual thinking of suggesting a lesser dampening effect, such as using 90% KRACH and then adding that to 5% for each team, just based upon the feeling that even the weakest team should have at least a 5% chance - but that's just based upon gut feel, no data analysis.adamw
jfeath17
adamw
BTW - I can't read your charts, so I have no idea what they're telling me. If there's an English translation, feel free.
My one sentence summary is that the current KRACH probabilities are biased towards the higher ranked team, a very simple modification which would greatly increase the accuracy is to change the formula to P(A Winning) = .749*(KRACH_A/(KRACH_A+KRACH_B)) + .126
Wait, that's English?
Nevertheless, I do appreciate and enjoy the KRACH model, and do believe it's the fairest way of ranking teams.
___________________________
Jeff Kahn '70 '72
Jeff Kahn '70 '72
Re: 2018 ECAC Permutations
Posted by: Swampy (---.ri.ri.cox.net)
Date: March 03, 2018 10:44PM
adamw
What future probabilities of any kind are verifiable?
Well, if we have a fair coin and flip it repeatedly, we can calculate the probabilities before they happen.
This hinges on the coin being "fair." We can decide this by measuring its dimensions, the smoothness of its surfaces, and the uniformity of its density and metallurgic composition.
Of course, none of these measurements can be without some error. But if the errors are not biased high or low, this shouldn't matter.
We believe dimensions, smoothness of surface, and uniformity of density and metallurgy are important because we understand gravity, the mechanics of flat bodies, etc. We understand such things they've been studied in a wide variety of contexts, not only coin flipping, and these understandings have been corroborated by experimental results. Moreover, laboratory experiments are "closed" in the sense that researchers control not only the conditions for observations but also can intervene actively in experiments to create desired conditions for observation (e.g. throw coins with varying strength).
In contrast, there are two problems with hockey games. 1) We do not have a credible theory of hockey that explains why teams win or lose. 2) Games occur in "open" settings, where what goes on is not under experimental control and contingent on things that themselves may have any influence.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: abmarks (---.hsd1.ma.comcast.net)
Date: March 03, 2018 10:49PM
jkahn
Basically, what's being suggested here is to use KRACH for 75% of the probability and split the other 25% equally. I was actual thinking of suggesting a lesser dampening effect, such as using 90% KRACH and then adding that to 5% for each team, just based upon the feeling that even the weakest team should have at least a 5% chance - but that's just based upon gut feel, no data analysis.adamw
jfeath17
adamw
BTW - I can't read your charts, so I have no idea what they're telling me. If there's an English translation, feel free.
My one sentence summary is that the current KRACH probabilities are biased towards the higher ranked team, a very simple modification which would greatly increase the accuracy is to change the formula to P(A Winning) = .749*(KRACH_A/(KRACH_A+KRACH_B)) + .126
Wait, that's English?
Nevertheless, I do appreciate and enjoy the KRACH model, and do believe it's the fairest way of ranking teams.
You are implying that there was a subjective choice of dampening effect?
Isn't the suggested formula there because it's the equation that defines the result of the logistic regression between KRACH and actual win% (when KRACH is grouped into buckets)?
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: jkahn (---.hsd1.il.comcast.net)
Date: March 03, 2018 11:05PM
No, all I'm saying is that my gut feel is subjective.abmarks
jkahn
Basically, what's being suggested here is to use KRACH for 75% of the probability and split the other 25% equally. I was actual thinking of suggesting a lesser dampening effect, such as using 90% KRACH and then adding that to 5% for each team, just based upon the feeling that even the weakest team should have at least a 5% chance - but that's just based upon gut feel, no data analysis.adamw
jfeath17
adamw
BTW - I can't read your charts, so I have no idea what they're telling me. If there's an English translation, feel free.
My one sentence summary is that the current KRACH probabilities are biased towards the higher ranked team, a very simple modification which would greatly increase the accuracy is to change the formula to P(A Winning) = .749*(KRACH_A/(KRACH_A+KRACH_B)) + .126
Wait, that's English?
Nevertheless, I do appreciate and enjoy the KRACH model, and do believe it's the fairest way of ranking teams.
You are implying that there was a subjective choice of dampening effect?
Isn't the suggested formula there because it's the equation that defines the result of the logistic regression between KRACH and actual win% (when KRACH is grouped into buckets)?
___________________________
Jeff Kahn '70 '72
Jeff Kahn '70 '72
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: jfeath17 (---.bstnma.fios.verizon.net)
Date: March 03, 2018 11:08PM
jkahn
Basically, what's being suggested here is to use KRACH for 75% of the probability and split the other 25% equally. I was actual thinking of suggesting a lesser dampening effect, such as using 90% KRACH and then adding that to 5% for each team, just based upon the feeling that even the weakest team should have at least a 5% chance - but that's just based upon gut feel, no data analysis.adamw
jfeath17
adamw
BTW - I can't read your charts, so I have no idea what they're telling me. If there's an English translation, feel free.
My one sentence summary is that the current KRACH probabilities are biased towards the higher ranked team, a very simple modification which would greatly increase the accuracy is to change the formula to P(A Winning) = .749*(KRACH_A/(KRACH_A+KRACH_B)) + .126
Wait, that's English?
Nevertheless, I do appreciate and enjoy the KRACH model, and do believe it's the fairest way of ranking teams.
Yes this exactly. Much better of a simplification than mine.
Re: 2018 ECAC Permutations - ECAC tournament draw
Posted by: BearLover (---.nycmny.ftas.verizon.net)
Date: March 04, 2018 01:23AM
adamw
Actually, I think you're the only one conflating anything, because I never said "worse" - so I'm not sure where this is coming from.
adamw
It is not certain that looking at things beyond wins and losses is any better. Goal differential has major flaws, and might not mean much. Shot differential has its own issues, but could be a decent factor. Honestly, I'm not all that interested in things like goal and shot differential.
So you wrote an entire post taking issue with how I put words in your mouth that you said shot differential/goal differential is a worse predictor than wins/losses, when in actuality you were merely saying both are equally flawed? Then just replace "worse" in my post with "just as bad" and my points still stand.
Oh I'm fully aware I don't really know anything. I don't have a background in statistics or mathematical modeling (or swimsuit modeling). I just watch a lot of hockey and read a lot about hockey, and nothing I've seen or read suggests a predictive model based entirely on win % over a 30-game season is going to be very accurate. You/CHN have done a convincing job arguing that KRACH is built to be (almost?) the best model for measuring past success. It isn't built to predict the future, though.adamw
What has rubbed me the wrong way with your posts is your unnecessary (to me) vehemence against KRACH, the twisting of what I've said, and the high-level of self-confidence in what you're saying. A little humility is helpful here because no one really knows.
Edited 1 time(s). Last edit at 03/04/2018 01:49AM by BearLover.
Re: 2018 ECAC Permutations
Posted by: jtwcornell91 (Moderator)
Date: March 21, 2023 10:23AM
Swampy
abmarks
Swampy
Some things I'd want to add to the discussion:
2. Exactly how does variance play out in these methods. If Team A plays Team B, does the P[Team A or Team B wins] = 1.0? Suppose Team A has P[winning] = 0.6, and Team B has 0.4, but Team A is erratic (I'm looking at you Clarkson), while Team B is not. Does A's greater variance show up in the prediction?
Let's say we know that in the long run, A beats B 75% of the time. So, over 100 games, A wins 75.
What the P(A winning) does NOT tell you is which of those 100 games A wins. A could go 0-10, then 75-5, then 0-10 over the course of those 100.
Taking that back to the topic at hand, short term results (ie the 1 game result in a tournament) are going to vary a lot vs. the long-term percentage.
I understand this but was talking about variance in several other senses. I'll explain them here. WARNING: THE FOLLOWING IS QUITE WONKISH.
Assumptions
Assume two teams, Team C and Team H, belong to a 12-team league in which teams play each other twice during the season. So each team plays 22 league games. Also assume teams earn 0 points in the league standings for a loss, 1 for a tie, and 2 for a win.
Estimation Variance
For the moment, ignore ties. Any data-based estimate of a team's chances of winning a game can be thought of as a function. If pC is the probability Team C wins a game, then let pC be the estimate of that probability. So that:
(1) pC = f(data)
In other words, the estimated probability is a function of whatever data are used in the estimate. When we say "data," this includes the number of data points (sample size) used to make the estimate.
Now, if we know the mathematical properties of f() we may be able to derive, mathematically, an expression for the variance of pC, var(pC). Call this the estimation variance, a measure if the estimate's the precision.
If we do not know the estimating function's mathematical properties, we still may be able to estimate var(pC) using simulation and resampling techniques.
Game and Game-Series Estimates
Think of a single game as an experiment with two possible outcomes: "success" and "failure." For simplicity, assume we actually know the real probability of each, so we don't have to use estimates like (1). To think about this, just consider Team C for now.
Let:
p = probability Team C wins
q = probability Team C loses = 1 - p
Furthermore, to convert the results into a number, define a random variable, X = 1 for a win and 0 for a loss. This is well known as a Bernoulli Trial, and X has a Bernoulli Distribution. The variance of X is given by:
(2) var(X) = pq
In the present context, call this "game variance" since it is the variance related to the outcome of a single game.
We can also think of a "series variance", which is the variance associated with a team winning a series. To simplify the math, let's disregard the fact that some series end after a team has won the majority of games in the series (e.g., 2 out of 3), and just think of the number of wins in a series. Define a second random variable, Yn as the number of wins in a series of n games. If each game has the same probabilities of its outcome, then Yn is the sum of n X's. In other words, it has a binomial distribution, the variance of which equals:
(3) var(Yn) = np(1-p)
In both (2) and (3) the variance depends on the value of p. If p = 0, the variance is 0, and similarly for p = 1. The variance is at its maximum, 0.25, when p = 0.5.
It's important to note here that the variance depends on the underlying, real probabilities and is not a matter of estimation.
Comments on jfeath17's chart
Notice though that confidence intervals plotted around a curve like this, which is based on empirical data, themselves estimated from other data (as in Equation 1), have two sources of variance: Estimation Variance and Game variance.
- The chart shows a relation between the Krach and actual game outcomes. Because of the properties of Bernoulli and Binomial distributions, the variance necessarily decreases as p moves away from 0.5 and closer to 1.0. So we would expect better predictions to the right of the graph. But the graph is almost a straight line up to about p = 0.85 and then drops off slightly. Maybe this is due to a weakness in the Krach, which is not intended to predict outcomes. Or maybe that's why they play the game.
- The chart would be improved with confidence bands, which are sensitive to variance and graphically show how confident one should be about the fitted line.
Perfomance Variance
In addition to the above, we should consider the variance of a given team's performance. Some teams are reliable; others are erratic. This can be best explained with an example.
Suppose every one of the 10 "other" teams always scores exactly 3 goals in every game. Then if O is the number of goals one of these "other" teams scores, the expected number of goals is 3 (E[O] = 3), and the variance is zero (var[O] = 0).
Similarly, assume Team C always scores 4 goals when it plays. Then if C is a random variable equal to the number of goals Team C scores, E[C] = 4 and var[C] = 0.
We can see right away that over the season Team C will always win over the ten "other" teams, so just from playing them it will accumulate 40 points (10 teams, 2 games per team, 2 points per win).
But now consider Team H, which is more erratic. Let H be the number of goals it scores in any given game. Like Team C let Team H's expected number of goals be 4: E[H] = 4. But unlike Team C, var[H] will not be zero.
Instead, suppose H has the following probability mass distribution: P[H = 2] = 0.10, P[H = 3] = 0.15, P[H = 4] = 0.50, P[H = 5] = 0.15, and P[H = 6] = 0.10. So here we can see different results when Team H plays its 20 games against the 10 "other" teams: the expected number of losses is 2, the expected number of ties is 3, and the expected number of wins is 15. So when Team H plays the other teams, the expected number of points is only 33, unlike Team C's 40!
What about when Team C and Team H play each other? Even though both have the same expected number of goals, the variance of Team H means it will be expected to lose to Team C 25% of the time, tie Team C 50% of the time, and beat Team C 25% of the time. In each of their 2 games against each other during the regular season, 2 points is at stake. So Team H can expect 0.5 points from a tie (1 point x 0.5 probability) and 0.5 points from a win (2 points x 0.25 probability), or 1 point in total. Similarly for Team C. With both teams playing each other twice during the season, each expects to get 2 points. This makes sense, because they're evenly matched.
But in terms of total points in the league, Team C expects to have 42 points at season's end, but Team H expects only 35 points. Which is how things should be, because Team H sucks.
Notice here that the only difference between the two teams is their respective variances, but it makes a big difference. If we look more closely at games against the 10 "other" teams, we are much more confident that Team C will beat them, whereas we expect Team H to lose to some of them. This is why performance variance is also important in thinking about which teams are likely to win particular games. Again, here there's no estimation issue. We know what the probabilities really are, yet variance affects the outcome.
Technical Suggestion
Jfeath17 asked for suggestions regarding the graphical analysis. For this kind of work I highly recommend the R Project's free, open-source statistical software used in conjunction with the RStudio GUI interface. It would allow easy addition of things like confidence bands in the probability plots, weighting of recent time-series data, etc.
So I think I missed the depths of this discussion when it happened, although I did comment on another thread. But I was wondering if either of you happen to have this collected in a slightly more organized form than a forum post, like an RMarkdown document?
I think there's an obvious improvement on predictions, written up in [arxiv.org] , which starts with the posterior probability distribution for the Bradley-Terry ratings, rather than the maximum likelihood estimate, which is what KRACH is. But in the example looked at there, it doesn't change the probabilities very much.
Re: 2018 ECAC Permutations
Posted by: upprdeck (38.77.26.---)
Date: March 21, 2023 10:34AM
Its a complex thing for sure.
You get the data over the course of a yr and then create a system that could explain the results, but then the next yr teams have changed by 20-40% and you can throw many of the numbers out the window.
then you have to decide how much back-2-back games matter vs home ice vs injuries matter
Look at horse racing with 100 yrs of stats and results measured and people cant even pick winners at better than 30%
You get the data over the course of a yr and then create a system that could explain the results, but then the next yr teams have changed by 20-40% and you can throw many of the numbers out the window.
then you have to decide how much back-2-back games matter vs home ice vs injuries matter
Look at horse racing with 100 yrs of stats and results measured and people cant even pick winners at better than 30%
Sorry, only registered users may post in this forum.