Complex system in place to evaluate umpires

Complex system in place to evaluate umpires

BOSTON -- They all ask for accountability.

"You know what? We're all accountable in this business," said Tigers manager Jim Leyland, a member of Commissioner Bud Selig's Special Committee for On-Field Matters, after a May ejection. "All of us are accountable, and when I say all of us, I mean everybody that's involved in the game needs to be held accountable. That's exactly what needs to be done."

Managers and players alike want the umpiring system to be as correct as possible. Red Sox manager Bobby Valentine, rarely shy with opinions, would go as far as adding lasers on the dish.

"Someone in here develop the way to call balls and strikes without that stupid idea of having human error," Valentine told the crowd at a seminar this month. "Who wants human error? I don't. So just get a plate with lasers.

"Seriously, figure it out. Have the lasers come up in front of the plate, have the lasers come from the side for the height of the batter. When it passes through, have the umpire know it passes through and have them go, 'Strike!' And when it doesn't, it's a ball."

Maybe someday.

In the present laser-less world, there are other means of making sure umpires are getting it right as often as possible. A Major League Baseball assessment system, not that widely known, uses on-site supervisors, video reviews, semi-annual written evaluations, high-end technology, incentives like playoff money and, in rare cases, suspensions to keep track of how umpires are doing.

"There is accountability that these systems ... are a perfect example of," said Randy Marsh, Major League Baseball's director of umpires after 28 years of calling games in the bigs. "Do we automatically send a guy to the Minor Leagues? No."

The daily reviews

Major League umpires and supervisors are probably the only ones who go to a game and don't care about home runs or great catches -- as long as they were actually home runs and catches.

The league employs six umpire supervisors and about a dozen observers covering 15 cities. Their jobs are to watch games in person, from the park, and record what they see. The difference between the two positions: supervisors have contact with umpire crews, observers don't.

All the people filling those positions have either been Major League umps, pro umps or been involved with pro baseball. Both jobs require two reports to be filed after every game, and plays that are marked incorrect or inconclusive are reviewed further.

"One is an evaluation report which covers how [umpires] moved around on the field," Marsh said. "How the crew worked together, how they react to the plays, how they handle situations, if they hustle, their demeanor on the field, their focus every pitch. There are about 12 different categories.

"The other report is an analysis report, which is any time there was any close call, it has to be reviewed by the supervisor or observer that is at the game to make sure the call is correct, and it is itemized on that sheet."

After every series, the umpires are given reports.

Not every game can be covered in person. Since 2011, the unattended games have still been reviewed, in whole, as part of the Supervisor Umpire Review and Evaluation (SURE) system.

"Each of our crews are assigned to a particular supervisor," Marsh said. "Let's say I have the crew that's in Chicago tonight and there's not an observer there, there's not a supervisor there. I get an e-mail the next day saying that game needs to be reviewed. I will bring it up on the SURE system, and I will review every play they had in that game. So every game is covered, one way or another."

The SURE system also allows the league to track calls other than balls and strikes throughout the year: base calls, fair-foul calls, etc. Marsh said umpires get those right more than 98 percent of the time.

There's a whole other entity for balls and strikes: the Zone Evaluation system. That was put in every park in 2009 to replace QuesTec, and it measures practically every pitch called by an umpire; batted balls are discarded. Cameras record the pitch in flight more than 20 times before it reaches the plate.

As of mid-August, umpires were calling pitches correctly at an average rate of more than 95 percent, by the league's numbers. Marsh said about a dozen umpires had called every pitch in a game correctly.

Only rarely is a pitch an umpire calls not calculated in the Z-E score.

"Occasionally, the catcher's movement is taken into consideration, as, at times, the catcher completely blocks out the umpire by moving up or sideways," Marsh said. "Usually, those are bad pitches anyway, but sometimes they are close."

The day after every game, home-plate umpires receive a breakdown of how they did.

"It will tell whether a pitch is 2.6-7 inches outside or 3.4 inches inside, or how many low or how many high," Marsh said. "It will tell them everything about that pitch, plus give them video of it, a graphic of it, where the pitch crossed the plate or didn't cross the plate."

Umpires themselves can review any play or pitch they want immediately after a game finishes. Their locker rooms league-wide are mandated to have video equipment for that purpose.

Incentives, corrective measures and philosophy

Besides the day-to-day looks, umps get a thorough vetting after both halves of a season.

"Mid-year evaluations go over everything they accomplished in the first half of the season, their Z-E scores, any plays that stood out -- any plays that they may have missed are itemized in that," Marsh said. "Any plays where they are assigned -- we call it 'does not meet,' meaning something happened on the field that they did not handle appropriately. There's a red mark and a 'does not meet' and a complete explanation of why they got it. All those things are on their mid-year evaluation for them to review. Every guy has to sign off on it; they have to read it and sign off on it. They go through the whole thing at the end of the season as well."

A tangible motivation for umpires to perform well is postseason money. Only the best umpires work the postseason, and that's based on all of the accountability work put in ahead of time. Making it to October baseball is a major goal.

In rare cases, an in-game mistake can get an umpire suspended. Bob Davidson was the most recent case, suspended for one game earlier this season because of a verbal spat with Phillies manager Charlie Manuel. Three umpire supervisors were dismissed following some controversial calls in the 2009 playoffs.

Still, players and managers sometimes ask for more disciplinary moves just like those. They can be demoted or cut, so why can't umpires?

"He probably needs to go back to Triple-A," Phillies closer Jonathan Papelbon said in June of D.J. Reyburn, a callup umpire. "You're up in the big leagues to do a good job, and when you don't do a good job, you should be demoted or fired."

Mike Port, the former vice president of umpiring for Major League Baseball and a former general manager, has a good one-liner on the subject.

"One thing that is just a personal irritation is when I see players say, 'Umpires need to be more accountable. [When] we screw up, we get sent to the Minor Leagues,'" Port said at the same seminar Valentine attended. "Well, having been on the club side, I can tell you, not necessarily so."

The league's logic is that it takes so long for an umpire to get to the Majors, if he wasn't capable at a high level, he would have been weeded out long ago. There are 68 full-time umpiring spots, with about 12 rovers who fill in from Triple-A. About 25 umps from Triple-A go to big league Spring Training.

The league does look at umpires more like players in a different manner. When someone is missing a call with some consistency, there's usually a reason. Umpires talk about opening up the outside corner just like a hitting coach does.

"We put in a lot of time, a lot of effort, a lot of money to groom these people, to be in the Major Leagues," Marsh said. "Just like a player goes through stretches where maybe he's not hitting the ball, well, why isn't he hitting the ball? For some reason, he's dropped his hands down, he's doing something mechanically wrong."

"This guy missed six pitches out of 160-something pitches," Marsh continued, creating an example. "That's a great score. But all six of those pitches were, say, off the outside corner of a left-hand hitter, and I wanted to know, why were his only misses out there? So I'll watch him a couple times, watch the scores, watch him a couple times. I'll pick something up; I'll guarantee you. I talked to numerous umpires this year just about that type of thing."

The difference between a physical error and a mental error computes just as it does for the players. If a call is missed because a guy saw something wrong, the argument can be made that it's inevitable throughout a season. But if an umpire is out of position, for Port, that's more egregious.

"But it bothers me to say they're not accountable, because these umpires are their own worst critics," Marsh said. "If I missed a play, and I come off the field and I look at it and I missed it -- believe me, the only person it eats up the most is me. I want to know why I missed that play. Was it positioning, was it timing, did I call it too quick? I'll pick myself apart, and I know all of our guys do it."

There will still be loud critics, though, on the field and off, when there's a mistake. The human element does that.

"You've seen a lot of missed plays," Yankees center fielder Curtis Granderson told The New York Times in June. "They're humans back there. They're going to make some mistakes. But part of the game is, sometimes there have to be some consequences for it. As players, if we make mistakes, there are consequences for us. You get errors, you get pulled out, possibly sent down. Different stuff happens to us.

"There has to be a similar type of situation on the other end. The one thing I've seen consistently, the umpire can be as loud and as animated as he wants, and the umpire gets to stay in the ballgame."

The league watches for bad ejections, too. Port suggested it has become rarer for an umpire to rip off his mask and charge toward a dugout, enraged. Emphasized to umps is knowing what to control, how to control it and perhaps most importantly, what to overlook.

They're also told to check the lasers regularly.

"Anybody can kick out a player -- it doesn't make you a great umpire," Marsh said. "What we're looking for is something, like you just said, we're looking for the guy that can handle all these things and these stressful situations with somebody looking over your shoulder every time -- plus handling yourself and handling some grumpy people in stressful situations. Keeping yourself under control -- that's what I think separates the men from the boys and makes the type of person that we're looking for in the big leagues."

Evan Drellich is a reporter for MLB.com. Follow him on Twitter @EvanDrellich. This story was not subject to the approval of Major League Baseball or its clubs.