Mathematical Models

[adamw]

I don't understand more than 2% of what John said - but I think I get the gist .... If you read the KRACH Explanation on College Hockey News, it's basically my "English" translation of everything that John explained to me. That's my role.

I think John's remarks though are very welcome and go a long way in explaining why no "better model" magically appears when it comes to forecasting. There are competing opinions on the models, and more to take into account than what others would lead to believe. That's why I always push back when people vehemently insist KRACH is "wrong" and there must be a better way. (see other thread)

John - I think you had a student who wanted to work with me on a better model at one time - but I wrote to the student, and then never heard back. That might've been a couple years ago - maybe more, I don't remember, time flies. ... But if anyone is interested, please pass them along.

[Swampy]

That gives you a uniform distribution of winning percentages (the kth best team out of n beats the n-k teams below them and loses to the k-1 teams above, for a winning percentage of (n-k)/(n-1) = 1 - ((k-1)/(n-1)), not one that clusters around 0 and 1.

FYP. (-5%)

Wrong.  Read it again.  And don't tug on Superman's cape.

Wait. I just counted parentheses: your the original has two left parentheses and three right parentheses. How can this be correct, Kal El?

OK. Now I see what you're talking about. I think both The Elements of Style and The Chicago Manual of Style, would want something to separate the equation from the parenthesis marking the end of the parenthetical elaboration. Maybe put the equation on its own line, use brackets for one or the other, or at least leave some spaces between the two closing parentheses. No matter what, an equation like

       (n-k)/(n-1) = 1 - (k-1)/(n-1)

is making some strong assumptions about the precedence of the operators. Does the equation want [1-(k-1)]/(n-1) or 1-[(k-1)/(n-1)]? Some computer languages would interpret the equation one way (i.e. they use left-to-right precedence), and others would interpret it the other way (i.e. they give * and / higher precedence than + and - ). I understand that mathematics generally uses the latter, but there's nothing lost, except ambiguity, by adding brackets.

All this is complicated by the fact that we're writing on a forum that will interpret a parentheses preceded by a quotation mark as a smiley "))."

[Trotsky]

All this is complicated by the fact that we're writing on a forum that will interpret a parentheses preceded by a quotation mark as a smiley "))."

This happens remarkably often and was definitely a Bad Decision by the Bureau of Smiley Standards at OSF or wherever.

[Trotsky]

Oh, BTW, thanks for the link. Everyone on this list knows Harvard sucks, but it's nice every once in a while to be reminded that even popular culture knows how pretentious the assholes that go to Harvard can be.

That scene gets my award for "best scene in worst movie."

Barrett in the penalty box may be #2.

[jtwcornell91]

That gives you a uniform distribution of winning percentages (the kth best team out of n beats the n-k teams below them and loses to the k-1 teams above, for a winning percentage of (n-k)/(n-1) = 1 - ((k-1)/(n-1)), not one that clusters around 0 and 1.

FYP. (-5%)

Wrong.  Read it again.  And don't tug on Superman's cape.

Wait. I just counted parentheses: your the original has two left parentheses and three right parentheses. How can this be correct, Kal El?

OK. Now I see what you're talking about. I think both The Elements of Style and The Chicago Manual of Style, would want something to separate the equation from the parenthesis marking the end of the parenthetical elaboration. Maybe put the equation on its own line, use brackets for one or the other, or at least leave some spaces between the two closing parentheses.

You're right, most journal style guides tell you to change one level of parentheses to brackets in that case.

No matter what, an equation like

       (n-k)/(n-1) = 1 - (k-1)/(n-1)

is making some strong assumptions about the precedence of the operators. Does the equation want [1-(k-1)]/(n-1) or 1-[(k-1)/(n-1)]? Some computer languages would interpret the equation one way (i.e. they use left-to-right precedence), and others would interpret it the other way (i.e. they give * and / higher precedence than + and - ). I understand that mathematics generally uses the latter, but there's nothing lost, except ambiguity, by adding brackets.

The principle I learned on operator precedence is "multiplication takes precedence over addition, and put everything else in parentheses to be sure". ;-)

[abmarks]

The principle I learned on operator precedence is "multiplication takes precedence over addition, and put everything else in parentheses to be sure". ;-)

7th grade algebra class FTW here.

[jtwcornell91]

I don't understand more than 2% of what John said - but I think I get the gist .... If you read the KRACH Explanation on College Hockey News, it's basically my "English" translation of everything that John explained to me. That's my role.

I think John's remarks though are very welcome and go a long way in explaining why no "better model" magically appears when it comes to forecasting. There are competing opinions on the models, and more to take into account than what others would lead to believe. That's why I always push back when people vehemently insist KRACH is "wrong" and there must be a better way. (see other thread)

John - I think you had a student who wanted to work with me on a better model at one time - but I wrote to the student, and then never heard back. That might've been a couple years ago - maybe more, I don't remember, time flies. ... But if anyone is interested, please pass them along.

If it's the student I'm thinking of, he ended up looking more into the game theory of how to design schedules for college football assuming a reasonable rating system.  I'm planning to try to recruit another student soon, but I think it's likely to end up being fall at the earliest due to the semester schedule.  There are basically two ways that KRACH is over-certain: first, as the OP points out, it should have a prior that keeps expected winning percentages from running away to 0 or 1.  But also, even if everyone's rating is finite, they're still uncertain, and that uncertainty should be included in the estimated win probability.

[KenP]

It sounds like you are trying to replace a deterministic KRACH rating with a probabilistic distribution.  It wouldn't be a normal distribution instead relative to rating.  Higher ratings would have PDF weighted downwards and vice versa for near-zero ratings.  From there you could assess the uncertainty in absolute chance-of-winning statements.

[jtwcornell91]

It sounds like you are trying to replace a deterministic KRACH rating with a probabilistic distribution.  It wouldn't be a normal distribution instead relative to rating.  Higher ratings would have PDF weighted downwards and vice versa for near-zero ratings.  From there you could assess the uncertainty in absolute chance-of-winning statements.

Well, it's not a matter of non-deterministic randomness, but of uncertain knowledge.  KRACH is the maximum likelihood estimate of each team's strength based on their game results.  This estimate has some uncertainty associated with it, although the interpretation of that from the frequentist perspective underlying maximum likelihood is not directly translatable into a probability distribution.  But in the Bayesian framework, the game results give you a posterior probability distribution for the team strengths, and the width (and shape) of this distribution influence future predictions.

[KenP]

It sounds like you are trying to replace a deterministic KRACH rating with a probabilistic distribution.  It wouldn't be a normal distribution instead relative to rating.  Higher ratings would have PDF weighted downwards and vice versa for near-zero ratings.  From there you could assess the uncertainty in absolute chance-of-winning statements.

Well, it's not a matter of non-deterministic randomness, but of uncertain knowledge.  KRACH is the maximum likelihood estimate of each team's strength based on their game results.  This estimate has some uncertainty associated with it, although the interpretation of that from the frequentist perspective underlying maximum likelihood is not directly translatable into a probability distribution.  But in the Bayesian framework, the game results give you a posterior probability distribution for the team strengths, and the width (and shape) of this distribution influence future predictions.

My point is that a "perfect rating" would assess the true worthiness of a team vis a vis others. KRACH gets us as close as we can with the data but it still is not "perfect".  The Perfect rating for a team would be a random variable with mean/median of KRACH with a PDF that leans more towards the mean for extremely good or bad teams.

[Swampy]

It sounds like you are trying to replace a deterministic KRACH rating with a probabilistic distribution.  It wouldn't be a normal distribution instead relative to rating.  Higher ratings would have PDF weighted downwards and vice versa for near-zero ratings.  From there you could assess the uncertainty in absolute chance-of-winning statements.

Well, it's not a matter of non-deterministic randomness, but of uncertain knowledge.  KRACH is the maximum likelihood estimate of each team's strength based on their game results.  This estimate has some uncertainty associated with it, although the interpretation of that from the frequentist perspective underlying maximum likelihood is not directly translatable into a probability distribution.  But in the Bayesian framework, the game results give you a posterior probability distribution for the team strengths, and the width (and shape) of this distribution influence future predictions.

My point is that a "perfect rating" would assess the true worthiness of a team vis a vis others. KRACH gets us as close as we can with the data but it still is not "perfect".  The Perfect rating for a team would be a random variable with mean/median of KRACH with a PDF that leans more towards the mean for extremely good or bad teams.

If KRACH is a MLE, wouldn't a resampling estimate of the pdf's characteristics yield something like what KenP is advocating?

[billhoward]

https://www.wsj.com/articles/math-madness-college-hoops-fans-hope-to-say-rip-to-the-rpi-1520604000?mod=searchresults&page=1&pos=2

Hardcore basketball nerds go nuts this time of year, not because the NCAA reveals which teams will compete in the Division I men's basketball tournament, but because a controversial metric known as the ratings percentage index plays a significant role in the selection.

Don't think of it as March Madness. Think of it as Methodology Mania.

RPI has been used to rank men's college basketball teams since 1981, and that's the problem: It's outdated. It's crude. And more sophisticated measures exist.

But the RPI persists.

Sixty-eight teams play in the NCAA's premier basketball tournament, which begins next week and runs through early April. Thirty-two of those teams automatically qualify by winning their postseason conference tournaments; the other 36 schools are chosen by a 10-member selection committee that relies on "team sheets," a sort of report card, with RPI rankings and other data.

Back in the day, RPI was the only tool designed to help the committee make objective decisions.

"It was pretty impressive for its time," said Ken Pomeroy, a basketball analytics guru who now advocates ditching the metric. "Before that, there was no objective statistical process for selecting the field."

RPI combines three factors: a team's winning percentage, which counts for 25% of the score; the opponents' winning percentage, which is 50%; and the opponents' opponents' winning percentage, which is 25%. The sum of the three factors is the RPI, and the team with the highest RPI is ranked No. 1.

One criticism of the original formula was that it didn't acknowledge home-court advantage, even though teams tend to win about twice as many games at home as they do away. In 2004, the formula was tweaked so that now, home wins count 0.6, or just over half a win, while road victories count as 1.4 wins. The opposite is true for losses: Home losses count 1.4 while road losses count 0.6. Wins and losses at neutral sites each count as 1.

[continues (paywall)]

[marty]

https://www.wsj.com/articles/math-madness-college-hoops-fans-hope-to-say-rip-to-the-rpi-1520604000?mod=searchresults&page=1&pos=2

Hardcore basketball nerds go nuts this time of year, not because the NCAA reveals which teams will compete in the Division I men's basketball tournament, but because a controversial metric known as the ratings percentage index plays a significant role in the selection.

Don't think of it as March Madness. Think of it as Methodology Mania.

RPI has been used to rank men's college basketball teams since 1981, and that's the problem: It's outdated. It's crude. And more sophisticated measures exist.

But the RPI persists.

Sixty-eight teams play in the NCAA's premier basketball tournament, which begins next week and runs through early April. Thirty-two of those teams automatically qualify by winning their postseason conference tournaments; the other 36 schools are chosen by a 10-member selection committee that relies on "team sheets," a sort of report card, with RPI rankings and other data.

Back in the day, RPI was the only tool designed to help the committee make objective decisions.

"It was pretty impressive for its time," said Ken Pomeroy, a basketball analytics guru who now advocates ditching the metric. "Before that, there was no objective statistical process for selecting the field."

RPI combines three factors: a team's winning percentage, which counts for 25% of the score; the opponents' winning percentage, which is 50%; and the opponents' opponents' winning percentage, which is 25%. The sum of the three factors is the RPI, and the team with the highest RPI is ranked No. 1.

One criticism of the original formula was that it didn't acknowledge home-court advantage, even though teams tend to win about twice as many games at home as they do away. In 2004, the formula was tweaked so that now, home wins count 0.6, or just over half a win, while road victories count as 1.4 wins. The opposite is true for losses: Home losses count 1.4 while road losses count 0.6. Wins and losses at neutral sites each count as 1.

[continues (paywall)]

The weights accounting for home-court advantage are applied to wins and losses before calculating the teams' winning percentage. But those weights are not applied to the opponents' or opponents' opponents' wins and losses, another point of contention.

Because most of the RPI is based on the strength-of-schedule measures, some argue that a team with a poor winning percentage could still earn a respectable RPI by playing good teams, or, on the flipside, playing weaker opponents could perhaps unfairly reduce a team's RPI.

To help make up for the RPI's shortcomings, this year, for the first time, NCAA team sheets include five additional measures: two results-oriented metrics and three predictive metrics. The new measures are Michigan State University Assistant Athletic Director Kevin Pauga's index; ESPN's strength of record metric and basketball power index; Mr. Pomeroy's rankings; and Jeff Sagarin's ratings for USA Today.

While RPI, a results-oriented measure, is confined to wins, losses and strength of schedule, the basketball power index, as an example, is a predictive metric that considers factors such as the coach's past performance and player injuries and setbacks.

(To what extent predictive measures should be used to pick teams for the tournament is another debate.)

The March Madness teams, their seeds and the tournament bracket will be announced this coming Sunday, and it could be the last year that RPI, at least in its current form, plays a leading role.

"We are in process of evaluating a different metric to potentially replace RPI," said David Worlock, director of media coordination and statistics for the NCAA. "We hope to have something done prior to the 2018-19 season."

The move would please many analysts and hoops fans, who would be happy to say "RIP" to the RPI.

Write to Jo Craven McGinty at Jo.McGinty@wsj.com

[jtwcornell91]

It sounds like you are trying to replace a deterministic KRACH rating with a probabilistic distribution.  It wouldn't be a normal distribution instead relative to rating.  Higher ratings would have PDF weighted downwards and vice versa for near-zero ratings.  From there you could assess the uncertainty in absolute chance-of-winning statements.

Well, it's not a matter of non-deterministic randomness, but of uncertain knowledge.  KRACH is the maximum likelihood estimate of each team's strength based on their game results.  This estimate has some uncertainty associated with it, although the interpretation of that from the frequentist perspective underlying maximum likelihood is not directly translatable into a probability distribution.  But in the Bayesian framework, the game results give you a posterior probability distribution for the team strengths, and the width (and shape) of this distribution influence future predictions.

My point is that a "perfect rating" would assess the true worthiness of a team vis a vis others. KRACH gets us as close as we can with the data but it still is not "perfect".  The Perfect rating for a team would be a random variable with mean/median of KRACH with a PDF that leans more towards the mean for extremely good or bad teams.

If KRACH is a MLE, wouldn't a resampling estimate of the pdf's characteristics yield something like what KenP is advocating?

The standard error associated with the MLE, whether calculated analytically or estimated via resampling or some Monte Carlo method, is still a measure of the variability of the estimator, not the range of plausible values of the quantity being estimated.  I.e., it says, if my true strength is 100, what's the spread of KRACH values I could espect due to the randomness of game outcomes.  Or you could use it to generate a confidence interval, and ask, what's the range of true strengths or which the spread of probable KRACH values includes 100.  But what you really want is, given the game results, what's the range of plausible underlying team strengths, and that's the Bayesian posterior distribution.

[adamw]

If it's the student I'm thinking of, he ended up looking more into the game theory of how to design schedules for college football assuming a reasonable rating system.  I'm planning to try to recruit another student soon, but I think it's likely to end up being fall at the earliest due to the semester schedule.

I'll take any help I can get. Send them over whenever you/they can. Just make sure they already are equipped with your stellar knowledge of the topic at hand.