But as the Yankees outfielder looks at his team's roster two weeks before pitchers and catchers report to Tampa, Fla., Swisher is well aware that Pettitte's name is not listed. If Pettitte ultimately decides he's needed more at home than on a mound, Swisher will understand.
"I love Andy Pettitte more than anything, not only as a person but as a player as well," Swisher said. "I think he's done so much for this game and the New York Yankees organization. I don't know what it's like to have kids -- I haven't been fortunate enough yet -- but I know that he misses his family.
"In my opinion, if he chooses to ride off into the sunset, he's definitely earned that. We would love to have him back, but if he's got to go, he's going with glory."
In New York to receive a Thurman Munson Award at the Grand Hyatt hotel in midtown Manhattan, Swisher weighed in on a day when it appeared -- at least for a brief period -- that Pettitte might have quietly decided not to report to Spring Training on time.
Steiner Sports, a memorabilia company, announced on its website that Pettitte was scheduled to take part in a private signing on Feb. 15, the same day that the Yankees' pitchers and catchers are set to have their first workouts in Tampa.
The listing disappeared later in the day, and Steiner Sports executive vice president of purchasing and player relations Chris Amoroso said that the company -- not Pettitte -- had made the change.
"Andy didn't say not to do the signing," Amoroso said. "We're just going to wait to see what he decides. We have nothing planned."
Yankees general manager Brian Cashman has repeatedly said that Pettitte has told the team not to count on him, beginning with the evening of the Bombers' loss to the Texas Rangers in the American League Championship Series.
Cashman said last week that Pettitte has already decided to retire, but the team was prepared to offer him a contract -- reportedly $12 million for one season -- if he wants to return.
"He's made a decision. It's just if he changes his mind," Cashman said. "I wouldn't even say he's undecided. He's decided not to play. If he decides to play, I think that's the rub. The only thing is, I'm left to constantly talk about it because I have to fill a void in the rotation. The obvious area to ask about is Andy Pettitte."
Without Pettitte in the fold, the Yankees have made smaller moves to bolster a staff that was projected to have Ivan Nova and Sergio Mitre as the fourth and fifth starters.
New York has signed veterans Bartolo Colon and Freddy Garcia to Minor League contracts with invitations to Spring Training, supplementing a crop of young hopefuls like Andrew Brackman, Dellin Betances and Manny Banuelos looking to make an impression this spring.
"Everyone is making changes; so are we," Swisher said. "I think the changes may be for the better. Getting to Spring Training, I think guys are going to step it up a little bit."
Bryan Hoch is a reporter for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.