Colome, who was making his first trip to the hill of the season, was lifted after five scoreless innings and 60 pitches with a 2-0 lead that the bullpen would protect for the remainder of the game.
Karns, who had averaged 100.4 pitches in five starts prior to Sunday's outing, was also removed after five scoreless innings with the Rays ahead, 2-0.
He had thrown 66 pitches.
"It was a little ironic what took place on Friday night, but I think we value, with Karnsie, he's thrown the ball well. At that point, I felt that he'd given us exactly what we needed," said Cash, whose bullpen, which had compiled a stellar 1.44 ERA in its previous 43 2/3 innings, would allow four runs in three innings on Sunday.
"If there was a chance we jumped the gun an inning early, that's on me."
"Yes and no," said Karns when asked if was surprised to get the hook after a strong fifth inning in which he sandwiched called third strikes to Travis Snider and Everth Cabrera around a pop fly to center by Caleb Joseph.
"It was a good ballgame and they felt like they needed to make a move. They're in charge of making those calls and I'm just there to do my job and pitch."
The normally-reliable trio of Brandon Gomes, Steve Geltz and Kevin Jepsen gave up for four runs and seven hits in 1 1/3 innings. Geltz took the loss while Jepsen, whose blown save was just the second by a Rays reliever this season, surrendered three hits without recording an out.
"We actually had the guys we wanted out there," said Cash. "[The Orioles] have a very good lineup and they put together some good at-bats, drew some walks, had some big hits, some timely hits.
"As far as our guys, all of them came out of the bullpen very comfortable with who they were facing and with the matchups," he added.
"From our end, it was just constant trying to stop the bleeding," Jepsen said. "After the fifth, it just felt like an uphill battle to just keep them off the bases. These have been the games where we've been able to come in and get out. For that not to happen today is unfortunate."
Cash finished his postgame comments with an observation that would likely generate no disagreement from any of the 16,652 fans in attendance at The Trop.
"It was just one of those days that the opposition handled us," he said.