For him, that included spending precious time with his grandchildren, talking mostly about fishing, maybe a little about hunting and golf.
Never about baseball, though. So imagine Erik Simpson's surprise when he read Gordon's obituary in a newspaper shortly after Gordon's death in 1978.
"I didn't know he was a famous baseball player until the day he died," said Simpson, who referred to himself Saturday as "the No. 2 grandson."
The late Joe "Flash" Gordon, who had eight grandchildren, will reach the pinnacle of his chosen game's fame when he is posthumously inducted into the National Baseball Hall of Fame on Sunday afternoon.
"I think he'd be very proud, but also very humbled by the experience," Simpson said.
Elected by the Veterans Committee in recognition of an 11-year playing career with the Yankees (1938-43, 1946) and Indians (1947-50) that was interrupted by his service in World War II, Gordon made nine trips to the All-Star Game, won five World Series rings and was named the American League's Most Valuable Player in 1942 -- beating out Ted Williams, who won the AL Triple Crown.
But unless you played with or against Gordon, a lighting-quick second baseman who led the AL in assists four times and still owns the league record for career homers by a second baseman (246), you'd have to rely on those fortunate men to explain his impact on the game.
Gordon also managed four teams and was involved in the only manager-for-manager trade in history, being sent from the Indians to the Tigers in exchange for Jimmy Dykes in 1960, but never was much for spinning any yarns about his diamond days.
Not even with his children.
"He was a very humble man," said Gordon's daughter, Judy, who will deliver the induction speech. "He never cared about his statistics, how many home runs he hit, any of that. He was just thrilled to be a part of baseball."
And the Hall of Fame is thrilled to have Gordon among the enshrined. Bobby Doerr (Class of 1986) and Bob Feller (Class of 1962) called Judy upon hearing the news in December.
"I also got several cards from Hall of Famers," Judy said.
Moments later, Tony Kubek, this year's recipient of the Hall's Ford C. Frick Award for excellence in broadcasting, walked up to Judy and gave her a congratulatory kiss.
Simpson suggested that Joe Gordon might have been "embarrassed a little bit" by all of the attention Hall of Famers receive on induction weekend. Judy Gordon, who watched her father play with Joe DiMaggio and many other legends at Yankee Stadium, suggested her father's selfless attitude was shaped by his less-than-privileged youth.
"He came from very poor beginnings," she explained, adding that her father at one time lived in a tent in the gold camps of early Arizona.
Providing a fuller sketch of her father's life is Judy Gordon's primary goal Sunday -- provided she can get through the speech. She's been working on it for some time, but it's not easy reaching the finish line.
"I cry really easy," she said.
Thus, she's taking no chances. She'll be wearing a rubber band on her wrist Sunday, and whenever the waterworks start, she'll try to, quite literally, snap herself out of it.
One problem, though.
"I've been wearing the rubber band when I'm practicing [the speech], and I've snapped it so many times I have blood blisters," she explained. "So tomorrow, I'll have to snap myself on the inside of my wrist instead of the outside."
Mychael Urban is a national writer for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.