Though Kelly got a no-decision, he made his presence felt by firing seven strong innings, allowing three hits and the one run.
"The only thing I want to do is just win," said Kelly. "That's what I do every single game. There's no hard feelings, and I don't think revenge is the right word there. I just wanted to go out there and pitch the best I can."
There were some good developments all around for the Red Sox in this tightly fought game.
Yoenis Cespedes made a terrific catch against the wall in left to help snuff out a rally in the sixth. Xander Bogaerts saved a run in the second, making a fine diving stop up the middle to take a hit from Matt Carpenter.
And in the ninth, it was Bogaerts who put the Red Sox ahead for the first time all night, lining a sacrifice fly to center against Cardinals closer Trevor Rosenthal on a 95-mph, first-pitch heater.
"I thought he wanted to get ahead of me with a fastball and go 0-1 so he could use his offspeed pitches," said Bogaerts. "I just went up there and geared up for the fastball."
Cespedes had started the winning rally with a single to left off Rosenthal. Mike Napoli then stepped up and delivered a big double into the corner in right, putting runners at second and third with no outs.
David Ortiz pinch-hit for Daniel Nava, but the Cards walked him intentionally.
Bogaerts did his job, then Koji Uehara did the rest, working around two singles in the ninth to notch save No. 23.
Boston's pitching staff had a stellar effort, finishing with eight straight scoreless innings.
Carpenter led off the first against Kelly with a double to right. With two outs, Matt Adams came through with an RBI single.
"There were some emotions there," said Kelly. "After the first inning, basically after the first pitch, I realized it was just pitching again. It's like pitching in the playoffs; you get really excited and anxious to pitch in the playoffs. Once you let go of that ball, it's just pitching again from there. It was definitely fun."
St. Louis starter Shelby Miller, Kelly's best friend, kept the one-run lead until the fourth, when Nava started a rally with a two-out single to right.
Up stepped Bogaerts, who lofted a fly ball to deep left. Matt Holliday got a glove on the ball against the wall but couldn't hang on. It was ruled an RBI double, and the score was tied at 1.
"I thought it was an out," said Bogaerts. "But it's a big ballpark here, man. You've got to hit it pretty good."
There was some concern for Kelly in the fourth when he hobbled a little after taking a liner by Oscar Taveras off his thigh, but he stayed in the game.
"It definitely hurt," said Kelly. "Definitely left some seams there and a little swelling. Just had to go out there. I wanted to pitch and didn't want to come out of that game at all. They just kept an eye on me to see if it was affecting my stuff. It was there every single time I let go of the ball. I just wanted to be in there. It's my first start for the Red Sox. I wanted to go out there and gut it out and do what I can to pitch."
The Cardinals threatened to take the lead in the sixth. After a walk to Holliday, Jhonny Peralta crushed one to deep left, but Cespedes ranged back and made a nice catch as he ran into the wall.
"I did actually get a pretty good read on that," said Cespedes. "I usually like to take a peek back at the fence to see where it is when I see the ball. But I wasn't able to do that this time, and that's why the fence kind of snuck up on me."
When Mookie Betts singled to open the eighth, manager John Farrell sent Will Middlebrooks to hit for Kelly. And in a somewhat surprising development, Middlebrooks was given the bunt sign. On the second pitch, he popped into a forceout.
"In a tie game, in a one-run situation, [we're] looking to advance 90 feet," said Farrell.
The Red Sox would finally get the run they needed in the ninth, and wound up snapping their three-game losing streak.
It was a strong first impression by Kelly for the Red Sox on a night he was swirling with emotions.
"Joe threw a great game," said Cards manager Mike Matheny. "We've seen him throw like that in the past, and we know he has that potential. That was a big outing for him, and even after he takes the ball off the knee, he was fighting and grinding. That's the competitor that he is. He didn't give us a lot. We had a couple balls hit hard, but we couldn't string a lot together. He did a lot of damage control."