Going into the eighth, Kelly had thrown just 81 pitches. He threw 16 more and was removed following a performance of 7 1/3 innings.
"I had a 10-run lead, but at the end of the day, it's something I've got to do better," said Kelly. "I wanted to throw a complete game really, really bad."
In the big picture, the Red Sox have found a pitcher they feel confident about for next season.
"I feel very good about Joe's presence in the rotation," said Red Sox manager John Farrell. "It's premium stuff. I think he's gained some consistency of the use to his fastball, particularly to the glove side of the plate. When you single out each of the pitches in his repertoire, it's premium stuff."
In the penultimate game of Derek Jeter's career, all eyes were on the longtime Yankees captain.
He went 1-for-2, and exited the game in the fifth following his infield chopper that went for a single in the third.
"Whatever he tells me," said Yankees manager Joe Girardi. "He told me two at-bats today, so that's what I did."
Jeter struck out on 99-mph heat in his first at-bat, the crowd chanting "Derek Jeter" for most of it.
"As anticipated," Red Sox manager John Farrell said of the affection for Jeter. "Our fans are great baseball fans, and they recognize a guy they've seen for 20 years here. To acknowledge the career he's had, it's not surprising in any way."
Jeter will have one last grand finale Sunday, not only playing once more, but being honored in a pregame ceremony hosted by his rival team.
For the Red Sox, Saturday was a good day.
Kelly scattered nine hits over his career-high 7 1/3 innings, walking two and striking out three.
After coming to Boston along with Allen Craig in a trade for John Lackey, Kelly went 4-2 with a 4.12 ERA in 10 starts with his new team.
"For any guy in the rotation, the ability to log innings is a primary goal," said Farrell. "His efficiency is because he's been able to pitch with his fastball and get some ground-ball outs."
Durability is something Kelly hopes to be known for in the coming years.
"It's definitely something that I want to do for the rest of my career. Eventually, I want to be a guy that throws 200 innings. To stay healthy and win more games than I lose," said Kelly.
The Red Sox came up with an eight-spot in the bottom of the second, sending 14 batters to the plate.
Rusney Castillo reached base twice in the lengthy frame, in which Yankees starter Masahiro Tanaka was removed with two outs. Christian Vazquez, Daniel Nava, Yoenis Cespedes (100th RBI), Garin Cecchini and Castillo all came through with run-scoring hits.
Much like Kelly, Cespedes has given the Red Sox a lot to be excited for heading into next season.
"He's clearly got middle-of-the-order presence for us," said Farrell. "I think the one thing we've been impressed with is the overall ability, the overall skills."
The 100-RBI milestone was a meaningful one for Cespedes.
"It's something very important for my career and it's something very special to me, to be able to get it in my third year in the league," said Cespedes. "There's only been seven guys to get to 100 RBIs this year. To be a member of that group is something pretty important to me and special."
Nava made it a 10-1 game with an RBI single in the sixth.
The Yankees had done close to no damage against Kelly until the eighth, when they struck for three runs.
"It was my last start of the year and there's no reason to hold back on the velocity or anything," said Kelly. "Other than that, it's the last time I take the hill this season and I decided to let it all out there."