The punchouts were all against established Major League hitters, as Kelly got Brett Gardner, Carlos Beltran, Brian McCann and Garrett Jones.
"He's got weapons to attack guys," said manager John Farrell. "He can run a four-seam fastball in. He can spread them out with both his changeup and curveball, which he did today. The overall quality of pitches thrown today was much greater."
Kelly went three innings against the Yankees, allowing three hits and two runs.
Of course, he wasn't going to break out his entire arsenal against a team he could face three or four times during the regular season.
"It's reading the game," said Kelly. "I got ahead, and if I was going to throw a regular start against these guys, I showed them my fastball, and I showed them my changeup. But I think I would have probably showed them more of my slider and curveball today if they started jumping on my heater and changeup like they kind of did in that third inning."
At the age of 26, Kelly is entering his prime years.
"I think when Joe's best, he's using his whole repertoire, because he can throw mid to upper 90s and not just be solely reliant on velocity," said Farrell. "I think when a pitcher gets into fastball mode, sometimes they start thinking more about velocity and they start overthrowing a little bit and sacrifice a little command.
"When he uses his breaking ball and his changeup, he buys some margin for error within the strike zone, and I think that has a chance for him to be more efficient with his pitch count. He's got such good stuff and it's just to trust a matter of trusting it on the plate, down in the strike zone."
Ian Browne is a reporter for MLB.com. Read his blog, Brownie Points, and follow him on Twitter @IanMBrowne. This story was not subject to the approval of Major League Baseball or its clubs.