Ogando Up, Tazawa Down: RE24’s Take on the Bullpen

Say what you will about ERA, but it conveys the story of a starting pitcher’s season very accurately. Just as we like converting position players’ offensive and defensive value into units of runs so we can weigh them accurately, ERA helps us put different pitching talents into perspective. What’s better, a wild, unhittable fireballer or a control artist who gets hit around? We don’t have to wonder how those translate into runs — ERA gives us a pretty good idea. FIP can tell us a bit more about how a pitcher’s walk, strikeout and home run rates should translate into runs allowed, but it’s just a retelling of the same story. BP’s new Deserved Run Average accounts for much more that we understand as contributing to hits allowed and runs scored.

The whole idea breaks down, though, when it comes to relief pitchers. There are a few who come in for clean innings almost exclusively, but once a reliever is brought in mid-inning even occasionally, ERA is not equipped to answer our questions. It comes down to the effect of baserunners, and the effect of partial innings.

Baserunners give ERA fits on the front and back end. On Friday, Alexi Ogando came in mid-inning in the 7th, allowed four baserunners on three walks, was lifted mid-inning after recording just one out, and ended up allowing just one earned run as Jean Machi got the third out of the inning (on a line out). For his efforts, Ogando saw his outing go into the books with a 27.00 ERA. That would be like allowing three runs per inning — and yet, if one allowed four baserunners for every one out, one’s ERA would end up quite a bit higher than that. And yet, had Ogando done even better than his ERA — say, he allowed four baserunners, but got two inherited runners out — we’d never guess it from his line in the box score.

Coming in mid-inning can throw even more of a wrench in things. On Saturday, Tommy Layne relieved Joe Kelly after the first out in the 8th, striking out Curtis Granderson, walking Yoenis Cespedes, and finishing off the inning with a Daniel Murphy pop out to second. Layne had himself a 0.00 ERA outing. But he did allow that walk — if he hadn’t come in with one out already recorded, what are the chances that Cespedes would have come around and scored? Not high, but not zero, either. Mix it in a cocktail with the irrelevance of inherited runners and the lack of control over the runners a reliever leaves behind, and ERA isn’t just a poor estimate of relief performance — it can miss routinely and wildly to one side or the other, depending on that pitcher’s normal usage and how his approach to pitching affects baserunners.

I’m not sure there’s a better way to evaluate relief pitchers than through run expectancy. Here’s the 2015 run expectancy table from BP, current through Saturday’s games:

Each of these numbers is the average number of runs scored in each of these situations so far this year (in over 2,000 games so far, but who’s counting), from the situation indicated through the end of the inning. So regardless of how an inning actually unfolds, we expect an average of .472 runs in each inning. Having a runner on second with no outs has led to an average of just over one run.

RE24 is a stat that makes use of these different run expectancies in each of the 24 different base-out states in the table above. You can calculate it for any player, but it really shows its muscle for relief pitchers. The stat is a counting stat, something you can accumulate over a season like strikeouts, so you have to keep a pitcher’s innings or games totals in mind, but I think it does everything we’d want in a relief pitching stat.

It’s a pretty simple calculation. If you come on with no outs and the bases loaded, well, we’re expecting some runs to score (2.246, on average). If the relief pitcher lets one runner score, but ties up the inning with just the one run, we’d subtract the one run from 2.246, giving the pitcher an outstanding single-game total of 1.246. But let’s say that pitcher did something we like — a ground out that resulted in a force play at home — but he had a runner reach. Now run expectancy has changed from 2.246 to 1.524. If he’s then pulled in favor of another reliever, he’s still reduced run expectancy, and his positive RE24 from that game (2.246 – 1.524 = 0.722) reflects that he did something positive, regardless of whether the runner he allowed (then on first base) comes around to score.

RE24 solves both kinds of baserunning problems. If the reliever who comes on with the bases loaded and no outs lets all three runners score, he’s done something we don’t like — and unlike with ERA, he gets penalized. If he’s lifted mid-inning, RE24 gives him credit for the exact situation he left behind rather than evaluating him through the next pitcher’s success or failure. In the Ogando example from Friday, he started with a run expectancy of .252 (one out, no runners). When he was pulled, run expectancy was .689 (two outs, bases loaded). So in terms of RE24, Ogando gets dinged for the full run that scored, and for the negative change in run expectancy (-0.437). A -1.437 RE24 in one day is a poor total indeed.

But RE24 also helps express the partial innings problem. If you come in for a clean inning and finish it with no runs scoring, you get a nice little RE24 bump of .472. When Layne handled the last two outs of the seventh on Saturday, though, he earned an RE24 of .252. Whereas ERA rewards him for a clean two thirds of one inning, RE24 is conscious of which two outs it was. Based on what we’ve seen for run scoring in innings with no one on and one out, if baseball changed its rules and only had two outs per inning, run scoring wouldn’t be two thirds of the current total — it’d be more like just over one half. Pitching toward the end of innings is easier, at least from the perspective of ERA. Layne allowed Cespedes to walk with two outs, and had he been lifted right then, Cespedes probably wouldn’t have scored. But the chances would have gone up if Layne had walked him with no outs — why should Layne get the benefit of frequently coming in mid-inning?

In the end, RE24 frequently matches our own evaluation of relief pitchers in a way that ERA does not.

In the end, RE24 frequently matches our own evaluation of relief pitchers in a way that ERA does not. Some guys are more likely to get the double play with an inherited runner, and that’s still real value they’ve added for the team. Pitching involves choices, too. If a reliever can “go for the strikeout” with a runner on third and fewer than two outs, and if he happens to do that and do it well — that helps in the RE24 department, and it helps the team, too.

If you’re still with me, here’s the payoff: RE24 puts part of the Red Sox relief leaderboard on its ear. Minimum 20 innings pitched:

*I used Baseball Prospectus’s version of FIP for all but Wright and Barnes, which are from FanGraphs as they were calculated for relief only. RE24 totals also from FanGraphs.

Since RE24 is keyed to the average number of runs that are scored in each situation, it’s zero sum — there’s as much negative RE24 to go around as there is positive. Layne’s -0.66 RE24, for example, says that in his near-full season, he’s been near-average — quite a bit better than replacement level. And Koji Uehara is as good as we thought he was, and Matt Barnes as bad as his other statistics have advertised.

The surprises are Ogando and Junichi Tazawa. Ogando has had a very high home run rate and a very, very high left on base rate (89%, per FanGraphs). He’s also had much better success than expected on balls in play (.253 BABIP), and a fairly high walk rate (3.67 BB/9) to go along with a good-not-great strikeout rate (7.67 K/9). That makes him look lucky to an ERA estimator like FIP, which hangs an amazingly high 5.62 around his neck. Maybe he has been lucky, or maybe is just successful in an unusual way — after all, Ogando does have 460 career MLB innings to his name, and suppressing hits on balls in play is a trick he’s accomplished all his career (.265 career BABIP against), and his left on base percentage has tended to be high, too. Whether it’s sustainable or strange, and regardless of whether it’s partly a byproduct of how he’s been deployed this year, it’s happened — per RE24, Ogando has been an above average pitcher in the Red Sox bullpen (even with that -1.44 total on Saturday!).

Junichi Tazawa’s high strikeout rate, low walk rate and low homer rate get translated to the second-lowest FIP on the team. He’s got the second-lowest ERA, too, but just as FIP made Ogando look extraordinarily lucky, it makes Tazawa look cheated, by a fair margin (three quarters of a run). But not so on RE24, which spits out a below-average total for Tazawa. He’s been nails with respect to his own baserunners this season (albeit less so than he was earlier this season), but a little more forgiving when he inherits them. That may be why RE24 hasn’t been as impressed with his work this year. But just like with Ogando, BABIP also holds some answers: while FIP ignores Tazawa’s fairly high .329 BABIP, it’s very much in line with his career BABIP (.324), and maybe that’s not something to ignore.

Photo by Robert Mayer/USA Today Sports Images