Earlier this month, when the PECOTA projection system forecast doom for this year's Baltimore Orioles, we dug into its methods to answer why. We mentioned only in passing something else that might be relevant: The Orioles have "overachieved" relative to their projections five years running, and not by a little.
Maybe that's just what they do? Maybe it's impossible to talk about the Orioles statistically without considering the overachievement factor?
To answer that, it helps to know how they overachieved.
It's not the offense
Baseball Prospectus uses an all-in-one hitting metric called True Average, which puts a value on every offensive action and scales it all to batting average -- so that .300 is very good and .260 is league average. It uses this to predict how many runs a team will score.
The Orioles' offense has had some excellent unexpected offensive performances over the past five years: Chris Davis emerged from mediocrity to become an offensive star, Nelson Cruz and Mark Trumbo set career highs in home runs after joining Baltimore, journeyman Steve Pearce turned briefly into a star, and so on. But overall the Orioles have played almost perfectly to their offensive expectations over the past five years.
Their projected .262 True Average over that time is actually slightly worse than their projected .264 True Average. They have scored almost exactly as many runs as their projections said they would in every year but one:
2012: The Orioles were supposed to score 713 runs, and scored 712
2013: Projected to score 686, they scored 745
2014: Projected to score 686, they scored 705
2015: Projected to score 707, they scored 713
2016: Projected to score 752, they scored 744.
So while the stardom of Manny Machado, the apparent success of a swing-for-the-fences approach and the front office's hyperactive approach to under-the-radar midseason pickups make for nice narratives, the simpler truth is that the Orioles have been an OK offense that has performed almost exactly as expected. They've sprinkled in a little bit of clutchness to boost scoring slightly. But the answer to the PECOTA question is not primarily the offense.
It's not the starters
PECOTA also projects how many runs a team will allow. Here, we see the overachieving Orioles emerge:
2012: They were projected to allow 807 and they allowed 705
2013: Projected to allow 755, they allowed 709
2014: Projected to allow 713, they allowed 593
2015: Projected to allow 728, they allowed 693
2016: Projected to allow 813, they allowed 715
That's a whopping 401 runs that the Orioles were projected to allow that they didn't, accounting for the bulk of their extra wins. And yet the Orioles' rotation hasn't produced a single great starter, or even a single great season from a single starter. No Baltimore starter has received a Cy Young vote, won more than 16 games or produced a sub-3.00 ERA in the past five years. Only one (Chris Tillman, in 2013) has made an All-Star team, and only one (Wei-Yin Chen, 2015) has appeared in the top 10 of an ERA leaderboard.
Indeed, five Baltimore rotations projected to be pretty mediocre have turned out to be pretty mediocre. We can compare how many wins above replacement PECOTA projected for the Orioles' top six starters (by projected games started), and how many wins they actually got from their top six starters (by actual games started), to see this:
2012: 4.3 WARP projected, 5.6 WARP actual
2013: 4.9 projected, 2.8 actual
2014: 5.5 projected, 4.1 actual
2015: 3.2 projected, 8.8 actual
2016: 5.3 projected, 6.5 actual
This passes a sniff test easily enough: Orioles starters have the third-worst ERA in the American League since 2012. They have the second-fewest innings pitched in that time. These starters have benefited a little bit from a better-than-projected defense, but the answer to the PECOTA question is not the starting pitchers.
We've now accounted for most of what makes up a team's performance: Its position players and its starters. We've found, so far, a team that was almost exactly what it was supposed to be. That leaves, of course, the bullpen.
It's the bullpen! By a lot!
The Orioles, like all teams, are not the same in the first inning as they are in the seventh, or the ninth, or the 13th. The players in the lineup might change; the players available to pitch certainly change.
In the first six innings of games, the Orioles are an OK team. In the past five years they have scored 2,522 runs in those first six innings and allowed 2,466. If that comprised their entire team, they'd be expected to win about 51 percent of their games -- an 83-win club. They'd have quietly outperformed their projections, with hardly anybody noticing because they'd have never actually made the playoffs.
But from the seventh inning through the ninth, they have outscored their opponents 1,004 to 917, the run differential of an 88-win team. And in extra innings, they've outscored their opponents 93 to 32, the run differential of a 144-win team. They've actually gone 50-22, collecting 16 more victories than a typical team would in the same number of extra-inning opportunities.
That, of course, gets credited to the bullpen, and particularly those members of the bullpen who pitch regularly in high-leverage situations. And this is where the Orioles have overachieved.
If we look at the top five Orioles relievers each year by average leverage index -- a measure of how close the game is when the pitcher is on the mound -- we see that all but one of them had a lower ERA than PECOTA projected. All but one!
A few years ago, I argued that winning with a great bullpen was the most fleeting way to win. The reason: Even at a team level, reliever performance is extremely shaky, fluctuating wildly from year to year as pitchers rise and fall in relatively small samples. The year-to-year correlation for bullpen ERA was tiny. While this year's best starting rotation is likely to be very good next year, this year's best bullpen is likely to be around average.
In fact, I wrote that piece about the 2012 Orioles, who were winning behind a great bullpen at the time. Since then, they've continued to win, even as the relievers themselves change, from Jim Johnson and Pedro Strop and Tommy Hunter to Zach Britton and Mychal Givens and Brad Brach. (And, always, Darren O'Day.) They have obliterated that argument.
Their most high-leverage relievers have allowed, cumulatively, 193 fewer runs than PECOTA projected. Britton alone has allowed 56 fewer runs than projected in the past three years. And, compared to the runs allowed in the first, these runs are especially likely to swing the outcome of the game.
Among the many mysterious challenges of baseball is predicting, measuring and anticipating the way a team might be better than the sum of its parts. A projection system typically does not attempt it. It merely predicts player performances, and the summing of those performances tells us whether a team is "good" or "bad."
PECOTA has never thought the Orioles were particularly good. In a sense, it hasn't been that far off. The offense has been what it expected. The starters have been what it expected. The bullpen has been very good, but bullpens are inherently unpredictable, and it's probably prudent to avoid projecting any bullpen will carry a mediocre team to the playoffs.
In another sense, though -- the sense that matters a great deal more -- the Orioles have obviously been a very good team, because they've won. They didn't get lucky; they performed better in the closest moments. They might not have been projectable, but they weren't flukes.
You can call that a whiff on the projections' part. I prefer to give the Orioles credit for doing more with less.