With the regular season complete, it’s time to look back at the 2015 college football season. Just as teams reflect on their seasons to evaluate their performance, ESPN Stats & Information will do the same for its Football Power Index (FPI).
As a quick reminder, FPI is a forward-looking system designed to measure team strength. Although much of the attention is placed on FPI’s team rankings (one through 128), its ultimate goal is to accurately predict game and season outcomes.
We are often asked, “How well does FPI actually perform?” In the interest of complete transparency, below you will find a breakdown of FPI’s performance in the 2015 regular season. Overall, the 2015 season was a difficult year to forecast, but FPI did as well if not better than almost any other system or analyst.
FPI’s game predictions
Without accurate game predictions FPI’s season projections would be worthless. Over the previous 10 seasons, FPI’s game predictions had performed well - the FPI favorite had won 75 percent of FBS-only games - but we took steps this offseason to make it even stronger.
Entering this season, we updated FPI’s game prediction model to include distance traveled, rest and game type (bowl, regular-season or conference championship game), which are factors that only strengthened FPI’s prediction accuracy.
The end result? The team FPI favored won 78 percent of FBS-only games this season, which placed FPI in the top 15 percent of the 68 systems tracked at predictiontracker.com. Going back further, the FPI favorite has won 75.4 percent of FBS games since 2005, which is a better percentage than the Vegas closing line (74.7 percent) during that time.
We know FPI is not only judged by how it performs in relation to other computer systems, but also by how it does compared to human game predictions. Over the course of the season, we tracked our ESPN colleagues’ game predictions. This includes any prediction on College GameDay, SEC Nation, ESPNU’s The Edge and various halftime shows.
We compared the analyst’s game picks apples-to-apples to FPI. In other words, how did FPI’s favorites do only on the games they picked? FPI “beat” 11 of the 12 analysts (good job, Greg McElroy), and overall when picking against the FPI favorite, the analysts were 78-137 -- incorrect 64 percent of the time.
There were certain teams that FPI had a grasp on and others that baffled the system. There were seven teams, including Clemson, Notre Dame and California, for which FPI had the correct favorite in all of their games this season. Add in another 33 teams for which the FPI favorite won all but one game, and the system had a very good understanding of about a third of the FBS.
On the other hand, there were five teams -- Oregon, Duke, Arkansas, Georgia State and Georgia Tech -- that won as FPI underdogs or lost as FPI favorites in more than half of their games. Many of those teams entered the season with high preseason expectations and then fluctuated in their performances throughout the year.
As with any projection system, however, if the FPI favorite loses it doesn’t mean the system was wrong. If a team has a 55 percent chance to win, it is expected to lose 45 percent of the time. In fact, if every team with a 55 percent chance to win was victorious, the system is actually pretty inaccurate.
As can be seen in the chart on the right, FPI’s predicted winning percentages corresponded closely to the actual winning percentage of teams this season. For example, when FPI gave a team between a 70 percent and 80 percent chance to win, those teams actually won 74 percent of the time. FPI's game predictions were fairly accurate in terms of the confidence the system had in each favorite winning.
FPI preseason projections
For a second straight year, ESPN released a preseason FPI that was based on a team’s prior performance, returning starters, recruiting data and coaching tenure. Unlike in 2014, when the top teams in the preseason AP Poll made the playoff, this year was much harder to predict.
Every team to make the playoff ranked in the top 20 of preseason FPI, but with a number of those teams losing a significant number of starters on one side of the ball (Clemson on defense and Alabama on offense), teams such as Ohio State, Baylor and TCU were all expected to be stronger than the four playoff participants.
Again, rankings are fun to debate, but ultimately preseason FPI should be judged on each team’s projected win total (which accounts for team strength and schedule) and conference projections.
FPI’s preseason projected win totals were within one win of the actual win totals for more than a third of FBS teams and within two wins for nearly two-thirds of teams. There were a number of teams, however, that FPI over- or underestimated.
UCF (projected to win 6.4 games), Ball State (projected to win 7.1 games) and Georgia Tech (projected to win 7.1 games) were all teams that FPI overestimated entering the season. Some of these are understandable -- the Knights had won at least nine games in each of their previous three seasons and the Yellow Jackets were coming off an 11-win season and Orange Bowl victory. Vegas actually had a higher over/under win total for those two teams.
Iowa (projected to win 6.6 games), Houston (projected to win 6.7 games) and Bowling Green (projected to win 4.7 games) are all teams that FPI underestimated. FPI saw that all three of those teams ranked outside the top 60 in the FBS in overall efficiency in 2014, but each of those schools deserve credit for fantastic seasons.
In terms of conference projections, FPI set a high bar for itself after the FPI preseason favorite won seven of the 10 FBS conferences last year. This year was a much harder year to predict; among Power 5 conferences, only the ACC correctly identified its champion in its preseason media poll. Michigan State and Oklahoma didn’t even receive a single vote to win their conferences from the media. FPI gave each of those teams about a 15 percent chance to take home its conference -- small, but not zero as the media poll would suggest.
We aren’t afraid to admit our “misses” -- FPI underestimated Alabama. It saw that the Tide were returning three offensive starters (not returning a QB) and were set to play the toughest schedule in the nation. FPI gave Alabama a 5.4 percent chance to win the conference, which was at least a higher percentage than media darling Auburn (1.4 percent). FPI was basically in line with ESPN’s analysts preseason picks for the Big 12, Big Ten and Pac-12.
Overall, FPI had some wins and losses throughout this season, but its general prediction accuracy was strong. We are never content, though, and will strive to make FPI's predictions as strong as possible for future seasons.
Feel free to track FPI’s progress throughout bowl season and be on the lookout for FPI’s way-too-early preseason FPI for the 2016 season, which will be released shortly after signing day.