College Football Ranking Systems, Retrodictive Accuracy and Cyclic Triads

So, you know all about sports?

Then the concepts of "retrodictive accuracy" and "cyclic triads" should be familiar terrain to you.

As B. Jay Coleman informs us in his article, Minimizing Game Score Violations in College Football Rankings, Interfaces 35(6), pp. 483–496, ©2005 INFORMS:

"[R]etrodictive accuracy is equivalent to minimizing game score violations: the number of times a past game's winner is ranked behind the loser".

Unfortunately, this is not always possible, since, as Coleman informs us, we face the problem of "cyclic triads", where Team A beats Team B, Team B beats Team C, and Team C beats Team A.

Nevertheless, the amount of retrodictive inaccuracy in college football ranking systems is far greater than cyclic triads alone would engender, Coleman has found that even the current best ranking system from the standpoint of retrodictive accuracy still contains violations which are at least 38% higher than the minimum which can actually be achieved.

Coleman thus developed a minimum-violations ranking solution - MinV, which ranks football teams so that the minimum number of retrodictive violations occur. As Coleman writes:

"The minimum number of game score violations MinV identified for the NCAA Division 1-A college football seasons from 1994–2003 was far superior to the numbers that 58, 68, 75, and 93 different ranking systems (Massey 2004b, c) produced in the years 2000, 2001, 2002, and 2003, respectively. Massey did not report ranking violation statistics prior to 2000."

He then examined the college football games from 1994 to 2003 and applied his results:

"to determine whether the consensus top five teams in any of these 10 seasons would have been changed if one had enforced a minimum violations criterion."

Using his system, the national champion would not have changed, but the rankings of teams below Nr. 1 would have changed, in part, substantially:

MinV ranking (Actual Massey Consensus Ranking in Parentheses)

Year...Number 1...........Number 2............Number 3...........Number 4..........Number 5

1994...Nebraska (1)......Penn State (2)......Colorado (3).......Alabama (5).......Texas A&M (8)
1995...Nebraska (1)......Florida (2)...........Tennessee (3)......Florida St. (4)....Colorado (5)
1996...Florida (1).........Ohio St. (2)...........Florida St. (3).....Arizona St. (5)...Nebraska (4)
1997...Nebraska (1)......Michigan (3)........Florida (4)..........Florida St. (2)....Tennessee (5)
1998...Tennessee (1).....Ohio St. (2)..........Florida St. (3).....Wisconsin (4).....Florida (5)
1999...Florida St. (1).....Nebraska (2).......Va. Tech (3)´......Michigan (4)......Kansas St. (5)
2000..Oklahoma (1).....Nebraska (8)........Washington (6)...Miami (FL) (2)...Florida St. (3)
2001...Miami (FL) (1)...Tennessee (3)......Florida (2)..........Oregon (4).........LSU (9)
2002...Ohio St. (1)........Miami (FL) (3).....Georgia (4).........Oklahoma (5).....Texas (6)
2003...LSU (1)..............Southern Cal (2)..Oklahoma (3)......Georgia (4)........Miami, OH (6)

Table 3: Although the national champions would not have been affected, the remainder of the consensus top
five NCAA Division 1-A college football teams for the 1994–2003 seasons often would have been adjusted
had a minimum-violations restriction been enforced (consensus rankings are in parentheses). I used Massey’s (2004b, c) consensus rankings for 1996–2003 and the final Associated Press poll in 1994 and 1995. Massey did not report a consensus ranking for 1994or 1995.

Coleman also examined:

"whether the participants in any of the BCS national championship games from 1998 through 2003 would have been altered by a minimum-violations restriction."

The answer is yes, as this table by Coleman shows:

............Minimum
............number of........Actual BCS..................................Participants with
Year.....violations........participants................................MinV adjustments

1998.....41...................Tennessee, Florida St...................Tennessee (1), Florida St. (2)
1999.....58...................Florida St., Va. Tech.....................Florida St. (1), Va. Tech. (2)
2000....51...................Oklahoma, Florida St....................Oklahoma (1), Washington (4)
2001....51...................Miami (FL), Nebraska...................Miami (FL) (1), Oregon (4)
2002....50...................Ohio St., Miami (FL).....................Ohio St. (1), Miami (FL) (2)
2003....59...................Oklahoma, LSU............................Oklahoma (1), LSU (2)

Table 4: The Bowl Championship Series (BCS), begun in 1998, would have
selected different participants in its national championship games in 2000
and 2001 if it had adjusted its final ranking to adhere to the minimum
number of violations (actual BCS rankings are in parentheses).

But note that the retrodictive approach is not a cure-all for ranking systems.

As Coleman himself admits:

[T]he MinV ranking may not make the best predictions of future games; as Martinich (2002) demonstrated, retrodictive accuracy does not necessarily result in predictive accuracy. However, no prior researcher has assessed the relationship between these two perspectives, because none have developed systems that optimize the retrodictive component (as measured by violation percentage) for college football. MinV allows that as a future research objective.

The entire article is definitely worth a read here.