3/10 Polls

USCHO.com Division I Men's Poll
March 10, 2003

   Team      (First Place)    Record  Pts   Last Week
 1 Colorado College   (33)    26-5-5  593     1
 2 Cornell            ( 7)    24-4-1  561     2
 3 New Hampshire              23-7-6  516     3
 4 Ferris State               27-8-1  449     5
 5 Boston College             23-9-4  438     6
 6 Minnesota                  20-8-9  383     7
 7 Michigan                   24-9-3  308     8
 8 Boston University         23-12-3  293    10
 9 Maine                      24-9-5  282     4
10 MSU-Mankato               18-8-10  267     9
11 North Dakota               24-9-5  237    11
12 Harvard                    19-8-2  172    12
13 Ohio State                22-10-5  125    15
14 Denver                    20-12-6   54    14
15 Michigan State            21-13-2   47     -

Others receiving votes: Providence 27, 
St. Cloud State 22, Massachusetts 9, Dartmouth 8,
Minnesota-Duluth 6, Miami 2, Quinnipiac 1

Interesting how even this late in the season it can still differ so much with PWR.

Very surprised that while Maine dropped dramatically, there wasn't a coincident rise for UMass.

The difference isn't too surprising to me. For example: BU may be #4 in the current PWR but with 12 losses it's hard to vote them that high in a poll. Maine's losses this weekend don't look nearly as bad in the numbers as they do to the pollsters. unny thing is, in those cases I think the polls have it right. Maine *should* be punished for tanking at seaons end and getting swept at home in their playoff series (they still deserve a bid I think, but definitely a low seed). Fifth place (in HE) BU doesn't deserve a top seed in the tourney, no matter what the SOS arguments are.

I think the differences simply point out real or perceived flaws in the selection criteria.

Rank School                             Pts (1st) Last  W-L-T
1.   Colorado College                   254 (16)   1   26-5-5
2.   Cornell University                 236 (1)    2   24-4-1
3.   University of New Hampshire        218        3   23-7-6
4.   Ferris State University            198        5   27-8-1
5.   Boston College                     183        6   23-9-4
6.   University of Minnesota            169        7   20-8-9
7.   Boston University                  137       10   23-12-3
8.   University of Michigan             133        9   24-9-3
9.   Minnesota State University-Mankato 118        8   18-8-10
10.  University of Maine                102        4   24-9-5
11.  University of North Dakota          97       11   24-9-5
12.  Harvard University                  66       12   19-8-2
13.  The Ohio State University           60       15   22-10-5
14.  University of Denver                32       14   20-12-6
15.  Michigan State University           24       NR   21-13-2

Others receiving votes: St. Cloud State University 8, 
Miami University (Ohio) 2, Providence College 2, 
University of Minnesota-Duluth 1.

Keith, I absolutely agree that the differences point out flaws in the selection criteria. In some cases, sportswriters and coaches who watch these teams every week are in a better spot to assess equitable rankings than are computer models.

I kind of pull for Miami of Ohio since a good friend of mine went there. But how the hell are they still getting votes? They haven't been good in months. Plus they lost to Sacred Heart. They proabbly got a #14 vote in both polls, probably by the same person (assuming the set of voters intersect).

[Q]Keith, I absolutely agree that the differences point out flaws in the selection criteria. In some cases, sportswriters and coaches who watch these teams every week are in a better spot to assess equitable rankings than are computer models.[/Q]
I absolutely agree that the differences point out the flaws of the pollsters. In no case can a sportswriter or coach follow all the teams in the country well enough to totally fairly distinguish a top 15. They would certainly not be aware of some games or of specific teams records versus the top teams. Plus, polls trend to a "what have you done for me lately" point of view. Had Maine had this loosing streak at the beginning, and finished up stronger, it would likely make a huge difference in the polls, with the pollsters all but forgetting about the early stuggles, while a mathematically system treats all games equally.



Post Edited (03-10-03 17:53)

I think Maine *should* be penalized for losing two at home to UMass in the playoffs. The PWR and RPI wouldn't change if the losses were spread out around the season, but I think that this is a flaw in the system. I would like to see a system where the playoff games are given more weight. I don't have a good suggestion, but I know what I'd like to see.

Delta, I agree with you that pollsters aren't objective and tend to put too much weight on what have you done lately. Mathematical formulas are objective. But they are only as "accurate" as the formula imposed. Flaws in the system are objectively carried through to the results.

It's perfectly reasonable to have debates on what the mathematical formulas should include and how they should work, I just don't think that the polls are a good place to start looking for what the goal should be . The polls have short-term memory in the season, and long-term memory for a school. UMass has had a pretty good season, beats Maine twice, and doesn't even get any votes in the polls. I'm sure them having never been a big time school has something to do with that.

I don't know if playoff games should necessarily be given greater weight, but I do think that any ranking system should give an indication of how good a team is right now.

Consider the following scenario: Team A wins its first 25 games, then has two of its star players go down with season ending-injuries and loses its last 5 games. Team B loses its first 5 games while waiting for its star goalie to come back from an offseason injury, then goes on to win out its remaining 25 games. Which of these teams deserves a higher seed going in to the tournament? I would have to say it's team B, who is playing well now, and is going to be a much harder team to beat.

It depends on whether you think the seeds are supposed to reflect who the committee thinks is going to win, or to reflect the team that has had the best season.

A system that treats game 1 the same as game 30 comes closer to giving a neutral evaluation of a team's season long performance and seems to me to be the better method for selection/seeding.

All this stuff with the recent memory was recently removed from the PWR rankings and the NCAA selection criteria when they chopped off the "Last 16" as part of the formula to compute your rating.

Is anyone here seriously advocating that we bring that back? Wasn't the reason they got rid of it because most teams play the final games in the season against teams from their own conference (which are sometimes very weak)?

Should a team which tanks at the end of a conference regular season be seeded lower in their conference playoffs than their overall conference record (which is determined by their play over the whole season) would call for?

KeithK '93 wrote:

> I would like to see a system where the playoff games are given more weight.

Amen to that.

To the tournament winners, playoff games mean automatic berths--->they turn out to be very important games. To everyone else, they count the same as the Anchorage-vs.-Fairbanks season opener--->ho-hummers. It's just not consistent.

Perhaps tournament performance could be considered in a manner similar to the mysterious "quality wins" factor in adjusting final RPI.



Post Edited (03-10-03 20:07)

[Q]I just don't think that the polls are a good place to start looking for what the goal should be.[/Q]

You're right, for all the reasons you cite. To elaborate on my position: the polls generally are subjective and not "acurate". So the differences between polls and ranking usually points to the flaws in the polling. At the present moment, I think it happens to highlight a couple of problems with the rankings (Maine and BU), at least from my point of view.

Naturally, choosing the ranking criteria is an entirely subjective process.

[Q]Consider the following scenario: Team A wins its first 25 games, then has two of its star players go down with season ending-injuries and loses its last 5 games. Team B loses its first 5 games while waiting for its star goalie to come back from an offseason injury, then goes on to win out its remaining 25 games. Which of these teams deserves a higher seed going in to the tournament? I would have to say it's team B, who is playing well now, and is going to be a much harder team to beat.[/Q]
What if Team A only lost their star players for 5 games, and they'll be back in town for the tournament... so we really create a mathmatical formula that starts figuring how how good a team will be based on who's in the lineup? That'd be an utter nightmare.

The greater important of the playoff games is the title. Otherwise it's just a game. I don't see why they should have to be given any greater importance than they already have. It doesn't make you a better or worse team if you lose to someone in the playoffs versus in the regular season. If we were to lose this weekend, does that really make us worse than if we had lost to RPI twice in the regular season, but beaten them in the playoffs?

I suppose the difference in philosophies is "it's the playoffs it should count more" versus "it doesn't make you better or worse depending on when you win/lose." It's one thing to have a general impression that there *should* be more weight, just because it's the post season, and if that's how you feel, fine, but I don't think that's just in as far as determine who's really the better teams.

There's a big difference between ranking teams for their conference tournaments and for the NCAAs. In the conference you have a round robin schedule where everyone plays the same schedule (or at least roughly). So using the entire conference slate and weighting all games equally is appropriate. For the national tournament, you are trying to decide, based on greatly divergent schedules, which teams have earned the right to play for the national championship. In this case, I think it is entirely appropriate to reward teams for success in their conference tournament (in addition to the autobids) and punish those who fail miserably like Maine.

My position, as I've said before, is that you should have to win something first (RS or tourney title) before having a chance play in the tourney at all. But then we wouldn't get to argue endlessly over chances for at-large bids and bubble teams, and therefore take away hours of February and March entertainment

The fact that star players were injured wasn't really central to my point. I included it to illustrate an extreme case where it is clear that a team is not as good going into the tournament as it was earlier in the season. Of course, I would apply the same logic to a team tanking down the stretch for any reason.

I do think, however, that seeding for the NCAA's should probably include some human discretion to account for known circumstances that the computer rankings don't capture (like injured players, etc.).

I don't get it... first, from what I can glean, you're saying that a team that loses its star player down the stretch, thus losing its last five games, should be ranked lower than one that lost its first five games. Now you're saying that the NCAA should be able to take injuries into account, thus possibly ranking them higher to compensate for the fact that they would be better if they hadn't lost their star player?

Wins are wins, and losses are losses. Teams are as good as their current roster, and they should be ranked and seeded by their performance on the ice, not by how good they'd be if their entire roster were healthy.

Beeeej

> ...but I don't think that's just in as far as determine who's
> really the better teams.

Well, that gets to the question of what the tournament bids and rankings mean. Are we trying to identify the "best" teams? Well sort of, but that's really an intractable problem. No ranking can really say with certainty whether BU is better than Minnesota in an objective sense. I say this for a number of reasons, of which the most significant is sample size. Another is the fact that, if there were an objective measure of "better" teams it would change significantly in time due to injuries or players maturing.

I see the tournament bids as identifying the teams that have "earned" a chance to play for the national championship. We have to some form of ranking system to determine this because there are at-large bids. But I think it's reasonable to more heavily weight conf. tourney games. This is mostly a feeling and it's hard to argue logically, but there it is.

Hey, if the conference tourney games should just be like regular games, why bother having the tourneys at all? The RS standings do a much better job telling you the relative strengths of the teams in a given league than any of the rankings could possibly do. Or than the tournament does. So why not just crown the champion now? We get the Whitelaw, case closed. After all, the reason they started playing the ECAC tournament in the first place was to select the NCAA teams.

[Q]Teams are as good as their current roster, and they should be ranked and seeded by their performance on the ice, not by how good they'd be if their entire roster were healthy.[/Q]

Everyone is completely misinterpreting my point. Here it is in black and white: "A team that sucks at the end of the season should be seeded lower in the end of season tournament than a team that sucked at the beginning of the season."

The whole thing about injuries was just to illustrate a potential reason why a team's performance might dramatically change mid-season. My point was exactly what you have said above... if a team's current roster is not playing well (for whatever reason), then they should be seeded lower than team who is currently playing well.

As to my latter point, I was merely suggesting that the committee (the humans, not the computers) be given some discretion in seeding. This might allow them to take into account a situation where some of a team's losses early or in the middle of the season occurred under extenuating circumstances (i.e., an injury) that no longer exist . This is in no way inconsistent with saying that a team that currently has injured players, and hence is not playing well now, should be seeded lower than a team that is currently playing well.



Post Edited (03-10-03 23:20)

Graham I:
[q]My point was exactly what you have said above... if a team's current roster is not playing well (for whatever reason), then they should be seeded lower than team who is currently playing well.[/q]

Graham II:
[q]I do think, however, that seeding for the NCAA's should probably include some human discretion to account for known circumstances that the computer rankings don't capture (like injured players, etc.)[/q]

If we're misinterpreting, Graham, I don't think it's entirely our fault. :-{)} Seriously - I'm not trying to cross-examine you, but how do you reconcile the two quotes above?

Beeeej

And yes, I realize the insurmountable impracticability of giving the committee such discretion, but it would theoretically produce a better field for the tournament.

Beeeej -- I think I edited my post to answer your question. If it doesn't, I agree that it's my fault. I do tend go on incomprehensible rants on this board from time to time



Post Edited (03-10-03 23:29)

From what I can see, your edited post now says that the committee should be allowed to adjust seeding to compensate for injuries earlier in the season, but should not be allowed to adjust seeding to compensate for injuries later in the season. How does that make any more sense than what you said before?

Beeeej



Post Edited (03-10-03 23:36)

It makes perfect sense. The goal of this discretionary system I propose would be to give teams that currently have the strongest rosters the highest seeds. Obviously injuries earlier in the season which are no longer affecting the team will be treated differently than current injuries which affect the roster that a team will be able to put on the ice during the tournament.

Example (assuming perfectly identical schedules for simplicity):

Team A: 25-5. Loses first 5 games due, in part, to injury. Players all back for the tournament.
Team B: 25-5. Loses first 5 games without "excuse."
Team C: 25-5. Loses last 5 games due to injury. Players all back for the tournament.
Team D: 25-5. Loses last 5 games due to injury. Players will not be back for the tournament.
Team E: 25-5. Loses last 5 games without excuse.

If these are the 5 teams in the tourney:

A and C are tied for 1st (losses under extenuating circumstance that no longer exists)
B would probably be 3rd (losses not "excused," but playing well down the stretch.
D and E would both be at the bottom because they are playing poorly now, and there is no reason (i.e., an injured player coming back) to expect that their play will improve at all in the tournament.


Now I realize this system could not be pulled off in any realistic setting, but I think it makes some sort of weird sense (though apparently only to me).



Post Edited (03-11-03 00:06)

The main flaw in your system, Graham, is that it relies too much on the good faith and good judgment of people who are only human. The reason most people on this board (from what I can see) favor objective systems is that subjectivity leads to controversy.

I agree that of your hypothetical teams, A and C appear to be the best. [STUPID HYPO ALERT] But what if the star player of team B just got dumped before the losing skid, but won his baby back in time to get the season back together? It isn't an "excuse", but it says as much about the team in November as the injuries to A and C do about their teams during their own 5 game stretches.

Teams lose for all sorts of reasons, and all teams have injuries big and small over the course of a year. Objective criteria will tend to average out the bumps that each team gets during the season (or at least over a number of seasons). Turning the selection process into the sort of excuse making that is the lifeblood of sports talk blather seems like a mistake. You want to bemoan your fate because of an injury at a bad time? Tell it to your priest, but as far as I am concerned teams A, B, C, D and E are the same at the end of the year. And trying to turn "games lost to injury into an objective criteria seems (1) impossible and (2) an even bigger mistake.

To take the most extreme example of what is wrong with your proposal, it would certainly be obscene to see a hypothetical 25-5 Merrimack team penalized for a first round conference loss because their goalie won't be available for the NCAA's.



Post Edited (03-11-03 00:29)

I for one think your system makes conceptual sense, but as you yourself said, ti would be an utter mess (well, you didn't quite say that) on an absolute scale. Plus, I don't like giving the committee discretion, when you have no way to know how and when they're gonna apply it.

The first proposal would be to RPI-ify or KRACH-ify Last 16. In order words, how well you did, weighted for SoS, over your last 16 games. But that wouldn't be entirely fair considering everyone plays more conference games near the end. Is it really fair to hurt us bc we have to play ECAC teams at the end, even if we schedule some good opponents in the first stretch of games? The only thing I could think might be fair is some kinda Last 16 versus how you shoulda done against those teams. In other words, have you fallen down or risen up, and it should probably only be a tiebreaking criteria, but the math is beyond me at this time of night .

-Fred

It is possible that the insular end-of-season conference schedules would not permit enough interconference play for KRACH to resolve itself or at least to make it a useful tool.

I agree that rampant subjectivity is certainly not something we want to inject into the system. However, I think there are certain circumstances in which computers just can't comprehend all of the subtleties that go into determining what makes one team better than another. Therefore, I think that the committee should be allowed to interpose some human judgment at the margins (I actually think they might already be able to do this to a certain extent, although the seeding criteria are not terribly clear).

I did not originally mean to advocate some sweeping system in which the only factor guiding seeding decisions was who was out with a tummy ache when. I merely suggested that if we are going to allow some degree of subjectivity, a factor like injuries might be something to consider. My more elaborate hypotheticals were not intended to advocate a fully discretionary system, but mainly to clarify and defend the internal logic of some of the statements I had made previously.

There is ultimately no cognizable difference between your thought experiment and "who had a tummy ache when." It can't be practically imposed without every other short-term flaw in a team being used as a back-room rationale for rank switching.

If you want to start with the premise that "we allow some subjectivity", then I agree that injuries can be part of the analysis. I do not, however, think that it is the committee's role to try and predict the winners of the tournament. It is to choose and reward teams that have had the best season, not the best March. In any subjective analysis of the effect of injuries, I would propose that the committee should not concern themselves with WHEN any losses due to injury occurred, but could give weight to THE FACT of losses due to an injury.


(How do you determine that a loss was "due to" a missing player anyway? By way of example, we certainly didn't lose in Estero because LeNeveu was at the WJC.)

[Q]In any subjective analysis of the effect of injuries, I would propose that the committee should not concern themselves with WHEN any losses due to injury occurred, but could give weight to THE FACT of losses due to an injury.[/Q]

This was exactly the point of including teams A and C in my hypo. They would be treated the same regardless of at what point in the season their losses occurred.


In my defense, having now laid on the table what is admittedly a seriously flawed scheme, may I refer everyone back to my posts that started this mess. These remain my points -- everything that followed was in defense of their internal consistency:


[Q]I do think that any ranking system should give an indication of how good a team is right now.

Consider the following scenario: Team A wins its first 25 games, then has two of its star players go down with season ending-injuries and loses its last 5 games. Team B loses its first 5 games while waiting for its star goalie to come back from an offseason injury, then goes on to win out its remaining 25 games. Which of these teams deserves a higher seed going in to the tournament? I would have to say it's team B, who is playing well now, and is going to be a much harder team to beat.[/Q]

[Q]The fact that star players were injured wasn't really central to my point. I included it to illustrate an extreme case where it is clear that a team is not as good going into the tournament as it was earlier in the season. Of course, I would apply the same logic to a team tanking down the stretch for any reason.

I do think, however, that seeding for the NCAA's should probably include some human discretion to account for known circumstances that the computer rankings don't capture (like injured players, etc.).[/Q]


In sum: (1) I think the end of the season should be given greater weight in the objective computer rankings (whether through last 16 or some new criteria), and (2) I think that a small degree of subjectivity might be acceptable at the margins because computers aren't perfect.


The point that the seedings should not reflect who is most likely to win the tournament is duly noted... I just don't happen to agree with it.



Post Edited (03-11-03 01:10)

BigRed Apple wrote:

> It is possible that the insular end-of-season conference
> schedules would not permit enough interconference play for
> KRACH to resolve itself or at least to make it a useful tool.

The way we dealt with this in KPWR was to use the KRACH ratings calculated with the full season's results as a measure of the strength of the opposition you played in your last 16 games.

DeltaOne81 '03 wrote:

> The first proposal would be to RPI-ify or KRACH-ify Last 16. In
> order words, how well you did, weighted for SoS, over your last
> 16 games. But that wouldn't be entirely fair considering
> everyone plays more conference games near the end. Is it really
> fair to hurt us bc we have to play ECAC teams at the end, even
> if we schedule some good opponents in the first stretch of
> games?

But if our opposition is easier in the last 16 games, we would be expected to win more of those games; the two ought two cancel each other out, allowing everyone to be judged fairly.

> The only thing I could think might be fair is some kinda
> Last 16 versus how you shoulda done against those teams. In
> other words, have you fallen down or risen up, and it should
> probably only be a tiebreaking criteria, but the math is beyond
> me at this time of night .

Here's what we did in KPWR: given the KRACH ratings of your opponents in your last 16 games, what KRACH would you need to have to be expected to win exactly the number of games you actually won out of those 16? That seems not entirely unlike what you're describing. For more details, see [slack.net]

Graham Meli '02 wrote:

> I think there are
> certain circumstances in which computers just can't comprehend
> all of the subtleties that go into determining what makes one
> team better than another.

If I see the word "computer" thrown out one more time to disparage objective consideration of a team's results, I may . What these unthinking computers do that you seem to have a problem with is look just at the actual outcomes of the games played. The last time I checked, the way that teams earned the right to play in the postseason was by winning games, not by having a talented team that would have won more games if not for inopportune injuries (or bad calls, or bad bounces, or any other reason you can come up with why a team shouldn't really have lost a particular game). Should we somehow get credit for the games in Florida because we were playing without our star goaltender? No, we lost those games, and that's how they go in the books.

The inherent problem with the human element/common sense/good judgement is that, even if the people involved are not overtly biased, the conclusions they come to will depend on the humans doing the judging. Then a team's playoff fate is not determined by how they performed on the ice (which is the only thing the "computer rankings" are taking into consideration) but the makeup of the committee. As far as I'm concerned, the less of that we have, the better.

DeltaOne81 said
[Q]The only thing I could think might be fair is some kinda Last 16 versus how you shoulda done against those teams.[/Q]

In fact, such a system does exist. The most widely used ratings system for chess (and some other games) is based more or less on (pardon my mathematical imprecision, and perhaps inaccuracy):

new rating = old rating + C (W - We)

Where C is a weighting factor corresponding to the importance of the game (if you think all games should be equally weighted, this number could be the same for each game).

W = {1 for a win, .5 for a tie, 0 for a loss}

We = the probability, using an established table, that you "should have" won the game, based on the difference between the ratings of you and your opponent at the beginning of the game.

Pros: Like the KRACH system (and unlike RPI), this automatically takes into account the difficulty of the opponents you've played, based on their rating. You never have to worry about your rating going down for beating a weak team. Unlike KRACH, it takes more heavily into account games that have been played more recently, without having the arbitrary threshold of Last16.

Cons: I don't think this system would be as good as the KRACH ratings at balancing out insular schedules (a persistent problem with RPI). However, the weighting factor could be controlled to make interconference games more important. Single-elimination tournaments would probably throw this system for a loop (although that also happened with L16). A problem this system shares with KRACH is that tends to predict that very good teams will almost always beat very bad teams.
While the average chess player could probably put up a good-faith effort against a grand master and lose hundreds of times in a row, I strongly doubt that (using KRACH's prediction), in a prolonged series against Rensselaer, Cornell would win roughly 17 games for each game it lost. Or that, in a Colorado vs. Mercyhurst game (for example, in the first round of the NCAA tourney this year), Colorado would be 65 times as likely as Mercyhurst to win. If Mercyhurst really stands only a 1.5% chance of winning such a game, then the current system of auto-bids is little more than a formality, or a scam to get MAAC fans to buy tournament tickets. But I think that hockey has enough random factors (due in part to low scoring compared to, say, basketball) like injuries and penalties that the probability of random, crazy stuff carrying Mercyhurst to victory would at least 1%. Going even further down, if Iona (which is only 1.5 games off of .500 in conference play) were to win their next 3 games (which KRACH suggests is not all that implausible), and face CC in a first round game, CC would be 215 times as likely to win. Think back over the last seven seasons of play for Cornell (or any other single team)--certainly there's been more than one fluke game.

Sorry for getting into a rant over KRACH (I really do like that system), but the point is that the chess ratings system actually makes the same sorts of claims. Obviously, I don't think that hockey should actually take up this ratings system. I just think that it's an interesting thing to look at, and a fun way to pass some time after a bye weekend.

I'm in favor of accounting for the last several games of the season (not sure that 16 is the proper number, but that is a mere detail).

Time is just as powerful a variable as any other you might use to rank teams. Over the course of a season, things happen. Injuries have been mentioned so far. But a laundry list of things can really CHANGE a team over the course of the season. Coaches get fired, off-ice distractions creep in, pressure to win builds, etc etc.

It seem perfectly reasonable to add a weight to end of season and/or playoff games. Just ask yourself who is more likely to win the tournament, the 25 win team that lost its last 8 games straight or the 20 win team who won its last 15 games.

In the scenario above, by NOT weighing late season games/playoff games, you'd likely give a lower seed to the 20 win team (assuming all other variables being equal for sake of this analysis). That's, for one, unfair to the OPPONENT of the 20 win team. Because everyone knows that they are a better team than the 25 win team AT THAT POINT IN TIME.

The NCAA tournament should be seeded to reflect accurately how good each team is ON THE DAY THE GAME IS PLAYED. This is the only way to ensure the equity of the match-ups.

Just what we need. Let the seeding committee use subjectivity. I can already hear the comments: They didn't pick us because they don't like us. Or, they picked them b/c they know their coach. Or any amount of permutations from the above. Right, just what we need. Since when does the selection committee have the right to decide subjectively? You think humans are less fallible than computers? People will find an excuse to complain regardless. No matter what happens, the selection process will never be "perfect." Injuries have nothing to do with the quality of a team. Hence, that is why it is a team. As a matter of fact, the longer a team is out with a star player, the more they should be able to adapt without him. And if they only win with him, then they aren't a very good team anyway, and they probably won't win the championship.

I promise this is the last thing I'll say about this, as the dead horse indicator on my desk has begun to flash:

Whether calculated by computers, PhD's with slide rules, or trained chimps, all purely "objective" ranking systems were originally created by humans who had to decide what factors to include the formulas. At some point some people made the subjective choice that OOper is relevant to deciding what teams are good and, say, goalie's save percentage is not. Therefore, any objective ranking system can be criticized as not properly capturing all of the correct factors. Save percentage may be a ridiculous hypo, but there certainly has been some reasonable debate here about whether recent games ought to be treated differently than early season games. That they are not is ultimately a human choice.

I have tried to argue, rather unsuccessfully, that there may be a whole range of factors that one might consider relevant in assessing the quality of a team that are not incorporated in the objective rankings. Injuries were just one possible example I posited, but are by no means the only, or most important, one. If we agree that a certain factor is relevant to determining how good a team is, there are two possibilities -- add it to the formula that computes rankings, or allow the committee to consider it "subjectively." Due to the immense difficulty of the former, I don't think the latter is always inappropriate. It might not be "fair," but some may argue that neither is a system that sometimes punishes teams simply for beating the weaker teams on their schedule.

Graham Meli '02 wrote:

> I don't know if playoff games should necessarily be given
> greater weight, but I do think that any ranking system should
> give an indication of how good a team is right now.

So it sounds to me like you would support a strength of schedule adjusted L16 factor in the PWR?

Personally, I really liked the L16 factor as I thought it did a pretty good job of giving credit to teams that got hot when it mattered. Sure, the lack of strength of schedule adjustment was a problem, but to drop the whole term was throwing the baby out with the bathwater in my estimation.

Ultimately, the thing that makes any deterministic system better than any subjective system is the determinism itself. Every team knows exactly what it needs to do to get in. If it does that which is logically necessary, it gets in. Don't underestimate the importance of that. Of course we all will have our own pet ideas of what constitute "fair" or "improved" criteria (mine are below), but those are hopelessly parochial and a matter of arguing at the margins. What does have significant meaning is knowing ahead of time that if you do what you need to you can't be screwed. That immediately transfers the onus of outcome from the criteria to the performance, where it belongs.

Tournament games already matter *far* more than RS games. They are only "ho-hum" if you lose. Solution: win.

L16 tried to capture something of value though the method was flawed. Bathwater problems: ignored s.o.s. and the 16 game cutoff was arbitrary. Ways to save the baby: progressively discount games as they get farther "back" on the schedule; factor in s.o.s. along with recency. Note that the current games vs TUC criterion has the same arbitrary cutoff problem.

The problem with ELO (the chess system mentioned above) is that it doesn't handle small numbers of results well at all. Classic case: I started out with an ELO of about 1750 because I was playing all my games against 1200-1500 players. Only when I got into the hundreds of results did my ELO fall into the real range for my skill, because by then I was playing opponents from all over the spectrum. ELO itself recognizes this: one's early rating is determined by a completely different algorithm that is similar to RPI. But this only works because at least the ratings of one's early opponents are usually "real" -- i.e., based on a significant number of outcomes. This just isn't appropriate for the hockey schedule, where everybody starts from scratch every Fall.

Oh, ELO does do something interesting, however -- the weight of your result against a team never changes after it is assigned. That means if you beat North Dakota early in the season when they were ranked high, you keep your mucho points for that even after North Dakota goes in the sewer later in the year. If, OTOH, you beat them later in the year along with everybody else, you get much less for your pains. There's something attractive in that, although of course the flaw can be seen from a mile away: what if North Dakota only looked good early because they had Canisius in a ball gag and leash? What's workable in a world which compares opponents with hundreds of results would lead to weird assymetries in hockey when everybody is bootstrapping early.



Post Edited (03-11-03 12:10)

Greg wrote:
> What's
> workable in a world which compares opponents with hundreds of
> results would lead to weird assymetries in hockey when
> everybody is bootstrapping early.

Conversely, you'd have trouble applying straight Bradley-Terry to world chess rankings, since you'd have to recalculate everything every time a game was played.

Should I summon Ken Butler to this discussion as well?

If you look at some of the KPWRs from past seasons, you'll also see that the L16 criterion KRACH is not terribly correlated with the overall KRACH, so you really can still reward teams that went on a tear late, even if they faced weaker opposition. I.e., playing your last 16 games against ECAC teams doesn't hurt you that mich if you go 14-1-1 in those games.



Post Edited (03-11-03 12:40)

Graham Meli '02 wrote:

> all purely "objective" ranking systems were
> originally created by humans who had to decide what factors to
> include the formulas.

But they decided before the season, when it wasn't known which teams would directly benefit from those choices. That's the main reason why many of us prefer objective criteria.

> It might not be "fair," but some may argue that
> neither is a system that sometimes punishes teams simply for
> beating the weaker teams on their schedule.

Which is why we're calling for RPI to be dropped in favor of a system that does a better job of accounting for strength of schedule.

Ah, had we only waited a bit, all our ratings problems would have been solved, presenting the latest from INCH:

Nine fun factors the committee should add to the selection process

1. Award points for universities ranked in Playboy's Top 20 party schools.

2. Have head coaches compete in American Idol-style talent show.

3. Cross off any team that doesn't offer free Internet Broadcasts of its games.

4. Bonus points for teams whose mascot can skate.

5. Multiply PWR by total inches of snow received since October 1.

6. Captains write essay: "Why my team deserves an NCAA bid."

7. Team buses gather at centrally located stock car track on "Run What You Brung" Night.

8. Supplement selection committee with tribunal of Judge Judy, Judge Ito and Judge Reinhold.

9. Ever see that Seinfeld episode "The Contest"?

> Conversely, you'd have trouble applying straight Bradley-Terry to world chess rankings, since you'd have to recalculate everything every time a game was played.

And I would build a time machine, beat Garry Kasparov when he was 5 (probably the latest possible age I could beat him at), and then 30 years later I'd be 1-0 against 2900 players.

How do tennis rankings work? Similar to ELO or RPI?

Another problem with ELO, completely unrelated to hockey - it's one of the computer rankings used in the BC$ (under the auspices of Jeff Sagarin).

I've always been curious where the BCS finds 7 different rating systems that don't account for scores. Is a Bradley-Terry computation among them?

The object of a sports contest is to win the game. In calculating the performances of teams (and we're talking about evaluating their performance during the season, NOT their potential or whatever else) wins and losses are the important thing. A rating system should restrict itself to those variables. Everything else is just somebody's opinion and subject to manipulation. So, the best thing to do is avoid that sort of thing.

I've always loved that I could evaluate college hockey and the NCAA selection probabilities with an analytical approach. That's what makes it far superior than listening to that loud, obnoxious bald guy make **** up.