The Giants tied the game at 3 in the top of the eighth inning on Brandon Crawford's RBI single that scored Buster Posey, who singled after a 10-pitch at-bat against Ramirez.
Colome had not given up a run in 19 straight appearances dating back to May 1, while Ramirez was working on a six-game scoreless streak.
"We go to two guys that we've leaned on and relied upon, and it just wasn't meant to be," Rays manager Kevin Cash said. "Erasmo looked like he tried to go away to Posey and it came back in, and tried to go in to Crawford and it went back more middle-away. Colome just threw a pitch that caught a lot of plate to Panik."
Colome said he tried to throw a cutter to Panik, but the ball stayed up in the zone and failed to break.
"It didn't do any rotation, like a fastball, no break," Colome said. "It's only one bad pitch. I don't have an excuse. I'm ready for tomorrow."
Moore said it was a surprise to see Colome falter in that situation.
"Obviously it's not very characteristic for him," Moore said. "'He's been lights-out. That pitch usually is something that's wipe-out for him, and for as many as he's thrown, to leave one right there ... We're all human. If there's a save situation [Sunday], he'll be out there at it again."
Moore gave up two runs on six hits in six-plus innings. He threw 106 pitches and struck out five while walking two. He pitched himself out of jams in the middle innings to keep the Rays ahead. In the fourth, the first two Giants reached base before Moore struck out Matt Duffy, Crawford and Angel Pagan to get out of the inning.
The lefty gave up a run in the fifth when he hit Panik in the helmet with the bases loaded and no one out. But Moore got a break when Brandon Belt hit a liner right to Logan Morrison, who doubled off Denard Span at second. Moore escaped the jam when Corey Dickerson made a sliding catch on a fly ball from Posey.
"There's nothing you can do about it," Moore said. "[Catcher] Hank [Conger] and I had some pretty good sequences to be able to keep the bat off the ball. I'm pretty happy with the way things worked out today."
Corey Long is a contributor to MLB.com and covered the Rays on Saturday. This story was not subject to the approval of Major League Baseball or its clubs.