And it all went up in flames when Bailey couldn't hold a 3-1 lead in the ninth.
His night started with Curtis Granderson lining a single to right. Raul Ibanez then came off the bench and clocked a game-tying two-run homer to right.
Ibanez later won the game, slapping a two-out walk-off single to left against Andrew Miller in the 12th inning.
"Threw it right down the middle," said Bailey. "There's nothing else to it. I left it middle and he did his job. There's nothing fun about going out there, letting guys run around the bases and do all that. Unfortunately, we'll have a long offseason. I'll have a long offseason to think about that.
"You've got guys like Pedroia going out there, playing with broken hands, broken fingers, whatever he has, getting that big double, Jonny throwing a [heck] of a ballgame, bullpen coming in and doing their job. I didn't. I've got to get back to just getting people out."
Opponents are hitting .370 against Bailey since Sept. 1.
The loss meant the Red Sox clinched their first last-place finish since 1992, when Butch Hobson was the manager.
"We didn't start the season to finish fifth or sixth, fourth or third," said skipper Bobby Valentine.
With the win, the Yankees maintained a one-game lead over the Orioles in the American League East heading into Wednesday night's regular-season finale. The Bombers win the division with a win over Boston. A New York loss coupled with a Baltimore win against Tampa Bay means the Yankees would have to play a one-game playoff to determine the division winner on Thursday at Camden Yards. The loser of that game would play in the AL Wild Card game.
"I'd just like to say congratulations to the winning team," said Valentine. "They battled hard and deserved to win."
Ex-Red Sox pitcher Derek Lowe earned the win for New York with two shutout innings.
Francisco Cervelli started the winning rally for New York, drawing a two-out walk against lefty Andrew Miller.
It's hard to fathom that Miller would walk Cervelli in that situation, considering it was the backup catcher's first Major League at-bat of the season.
"You know, I still got ahead of Cervelli there, but I didn't make pitches," Miller said. "I thought I made some good pitches for him to chase, but he walked and I kind of lost my rhythm for Granderson."
Granderson also walked, giving Ibanez the chance to end it. The lefty slugger poked an opposite-field single through the hole and into left, and Cervelli scored easily from second.
"I felt like I gathered myself and made some good pitches to Ibanez, but I killed myself by walking those two guys and putting them on, and it's not a good way to finish the game," said Miller.
Alex Rodriguez nearly won it for the Yankees in the 11th. After a two-out single to left by Nick Swisher, A-Rod clocked one to deep left-center, but Jacoby Ellsbury hauled it in against the wall.
"Yeah, I thought it was the same effort that everyone gave tonight," said Valentine. "They're doing everything they could to keep them from having more runs than us."
Bailey didn't settle down after the Ibanez homer. With one out, Derek Jeter lined a double into the corner in right. Boston elected for an intentional walk to Swisher. Instead of getting the double play he wanted from A-Rod, Bailey walked him to load the bases.
That was all for Bailey, and on came Mark Melancon to try to get the Red Sox out of the jam. Melancon did a nice job cleaning up the mess, inducing Mark Teixeira into a flyout to shallow center and getting Robinson Cano on a grounder to second.
Bailey went one-third of an inning, allowing three hits and two runs. It was his third blown save in nine opportunities since coming back from right thumb surgery.
"No one likes losing," said Bailey. "I've let the team down a couple times. It's my fault. There's no one else to blame. It's a mixture of not getting ahead of guys, making bad pitches when I've got to put people away. That's a situation where I've got to put people away, and I didn't."
Making his final start of the season, Lester turned in a strong performance, scattering eight hits but just one unearned run over five innings. The lefty threw 83 pitches, 48 for strikes. He had back spasms in the fifth, which cut his outing an inning or two short.
Lester finished the season 9-14 with a 4.82 ERA, taking a no-decision.
It was Pedroia -- fractured finger and all -- who put the Red Sox in front. After Ellsbury led the game off with a single, Pedroia lofted an RBI double into the gap in right-center.
"Yeah, he had good at-bats all night," said Valentine. "He's an amazing character."
Cody Ross got Pedroia home on a sacrifice fly to give Lester a 2-0 lead.
In the second, the Yankees got a run back, thanks to some help from the Red Sox. With two outs and nobody on, Granderson reached on an infield single and moved to second on a throwing error by third baseman Pedro Ciriaco. Eduardo Nunez followed with a grounder that shortstop Jose Iglesias tried to backhand, but he couldn't make the play as Granderson scored. It was ruled a hit.
James Loney gave Boston an insurance run in the ninth, drilling a solo homer to right, but that wound up not being enough.