Tampa Bay is 4-1 on the homestand, which has one remaining game -- on Wednesday night against the Yankees -- before the team takes off on a three-stop, 10-game road trip.
Price was making his fifth start of the season after going 0-1 with a 6.26 ERA in his first four. Although he struggled at times in those four starts, he looked sharp in this one.
Entering the ninth he had allowed just two runs and thrown just 92 pitches, but Robinson Cano singled to left to start the frame, prompting a pitching change.
Manager Joe Maddon explained the move by saying he thought Price would have been a little frustrated about giving up the hit, but had he gotten the out, he would have been pumped up to proceed.
Price was not happy about the move.
"Absolutely," Price replied when asked if he was upset about being pulled. "I felt like if I face one of the best hitters in the game of baseball, I can get the other guys out as well. I want to be in that game. I feel like that's my game to lose. And it stinks."
Fernando Rodney took over and struck out Vernon Wells for the first out of the inning, but Cano stole second on the pitch.
Rodney then intentionally walked Travis Hafner before Lyle Overbay drew a walk to load the bases.
"The big play was the walk to Overbay," Maddon said. "I thought Fernando had great stuff -- even the walk, the 2-2 count to Overbay before he walked him. The last pitch was pretty close, too, although I do think it was a ball. But I thought Fernando had great stuff. His velocity was 97, 98 easily tonight."
One out later, Ichiro Suzuki singled through the middle, driving home two and putting the Yankees up, 4-2.
"I haven't been able to do much, but hopefully things like this will get me past it and I'll have a good year," said Suzuki, who entered the game hitting .200. "Obviously, for any team, when you can beat their ace, it's a big win."
Rodney has now allowed four earned runs in seven appearances this season after allowing just five earned runs over 76 appearances in 2012.
Mariano Rivera allowed a leadoff home run to Evan Longoria in the bottom half of the ninth before finishing off the Rays to end the game.
Price is only the second reigning Cy Young winner in history to have his team lose his first five starts of the season, joining Minnesota's Frank Viola, whose Twins lost his first seven starts in 1989 while he went 0-5.
Despite pitching well, Price was not encouraged by his performance.
"We're still 0-5 when I pitch," he said. "I don't feel better."
Tampa Bay jumped out to a 1-0 lead thanks to Ben Zobrist's sacrifice fly in the first, and all looked well for the Rays, as Price posted three scoreless frames before striking out Eduardo Nunez swinging at a curveball to start the fourth.
Unfortunately for Price, the strikeout pitch turned into a wild pitch when the ball went to the backstop, allowing Nunez to reach first. Cano followed with a single to right, then Wells laced a single between first and second to score Nunez and tie the score at 1.
Jose Molina's RBI single in the seventh gave Price and the Rays a one-run lead, but New York answered when Suzuki singled and later scored when Brett Gardner grounded out to second.
Price has now surrendered a lead six times over his last three starts.
"When you're facing a guy like David Price, you're not going to get many opportunities, so you'd better take advantage," Yankees manager Joe Girardi said.
Maddon noted that if the Rays' offense had come through, things would have been different for Price.
"If he throws the ball like that and we had scored four runs tonight, he walks away and feels fantastic," Maddon said. "But tonight we don't score enough runs, and he doesn't feel as good about himself. That's just how this baseball thing works. ... I thought he was really good tonight. It was just unfortunate he didn't get the win."