But now, on the 30th anniversary of what has become known as the Pine Tar Game, Brett is smiling, even laughing, about what became the most memorable moment of his 21-year career with the Kansas City Royals.
Brett remembers every moment as he circled the bases on what appeared to be a go-ahead two-run home run off future Hall of Fame closer Goose Gossage in that July 24, 1983, game at Yankee Stadium. Then Yankees manager Billy Martin protested that Brett's bat was illegal because there was pine tar too far up the barrel.
Home-plate umpire Tim McClelland ruled in favor of Martin, called Brett out, erasing the home run and prompting an enraged Brett to storm off the bench in one of the few emotional displays of Brett's career.
And was it ever emotional.
"My boys weren't born yet, but every now and then, one of them will say, 'Hey Dad, can you play the video where you go nuts?'" said Brett, now the hitting coach for the Royals.
Not that Brett is complaining.
"You hit a home run in the ninth inning and give your team a lead, and you feel like you've accomplished something to help your team win a ballgame," said Brett. "Then, after sitting in the dugout, watching the umpire discussing the play and Tim called me out, yeah, I went a little ballistic.
"Tim was a rookie. Joe Brinkman was the crew chief, and he told Tim, 'You've got the plate, you call him out.'"
McClelland did. Brett reacted.
"The picture in my mind is George coming out of the dugout almost swimming," said Don Mattingly, the Yankees' first baseman, remembering how Brett's arms were swinging wildly as he charged from the dugout.
The Royals appealed. Lee MacPhail, then the president of the American League, ruled in favor of Kansas City. The final four outs of the game, picking up after the Brett home run, was played 25 days later, and the Royals emerged with a 5-4 victory on Brett's home run.
Brett? He was finally able to put behind him the hemorrhoid jokes that stemmed from his emergency surgery after Game 1 of the 1980 World Series against Philadelphia.
"From 1980 to 1983, everywhere I went, the drunks that were watching would make these hemorrhoid jokes," said Brett. "July 24, 1983, was the last hemorrhoid joke I heard. After that, it was pine tar.
"Would you rather be known as the guy with hemorrhoids or the guy that hit a home run off Goose, and then was called out for using an illegal bat, who was vindicated and the home run stood?"
No debate on Brett's part.
The bat meant enough to Brett that to get it back, he traded a bat he received during the 2011 Hall of Fame ceremonies that had the signatures of 60 Hall of Famers on it.
"[The man who had the bat] brought it over to my house this year and wanted me to sign it, and I was mesmerized by it," said Brett.
Over time, Brett has realized that the sporting public was mesmerized by the event, too. It is the moment he is most asked about regarding his Hall of Fame career, and for Brett, it was a big moment, as well, because he hit the home run off Gossage.
Brett only hit three other homers off Gossage. There was the three-run homer in the seventh inning of the Royals' 4-2 victory in the sweep-clinching Game 3 of the 1980 AL Championship Series. And there were two of his 317 career regular-season home runs -- a game-tying eighth-inning shot in an April 1992 game at Oakland that the A's eventually won 4-3 in 13 innings, and the Pine Tar shot.
"Because of the attention, everyone thinks I owned Goose, but I didn't," said Brett, who had a career .286 average against Gossage. "Truth is, he got the better of me."
Brett, however, eventually had the last laugh for that one he hit on July 24, 1983.
Not that it was easy.
Not with Martin managing the Yankees. Martin was always looking for an edge, and usually he found one.
Martin, after all, waited more than two weeks to protest the excessive pine tar on Brett's bat.
"Graig Nettles is the one who noticed the pine tar when we were playing in Kansas City, but Billy didn't say anything then," said Yankees reliever George Frazier.
Dave Righetti, also a pitcher on that Yankees staff, said, "Billy said we'd wait until we needed it. He wanted to wait for Brett to get a big hit that could impact a game."
The wait paid off, in big part because Brett was such a magician with a bat. Two weeks later, at Yankee Stadium, Brett was still using the same bat.
"I'd use bats for three weeks, a month at a time," said Brett, a career ,.305 hitter. "I didn't break many bats."
That, in fact, said Brett, was part of how the pine tar became an issue.
"I used a bat that was raw ash, and the problem was the pine tar would get inside the grain and start growing inside the bat," he said. "It was sticky. You used it to get a good grip on the bat, not to make the ball go farther."
The rule limiting the pine tar to within 18 inches of the handle wasn't written because it gave an edge to hitters, but rather because of a potential advantage for a pitcher on any ball that might pick up the sticky substance, which was why MacPhail ruled in favor of the Royals' appeal.
Martin, however, wasn't deterred. He still was looking for an edge when the Royals returned to Yankee Stadium to complete the game on Aug. 18, a scheduled off-day for both teams.
Martin first made it known he believed upholding the appeal was a joke. He inserted Mattingly, a left-handed-hitting first baseman, at second base. He put left-handed starting pitcher Ron Guidry in center field.
Frazier was picked to relieve Gossage and get the final out in the ninth.
"Before I threw a pitch, Billy told me to throw to first, second, third and home," said Frazier.
Frazier made the throw to first to appeal whether Brett had touched it as he rounded the bases 25 days earlier. Umpire Dave Phillips signaled safe. Martin was ready to argue.
"Billy comes out and says, 'How do you know that?'" remembers Bud Black, a pitcher with the Royals at the time. "He says, 'You weren't even here for [the home run].' Davey pulled out a piece of paper, a sworn affidavit from all the umpires, that said George did touch all the bases."
"Mr. MacPhail knew Billy would have something up his sleeve, and he was ready for him," said Frazier.
Frazier struck out Hal McRae to end the top of the ninth and then watched Royals closer Dan Quisenberry retire the Yankees in order in the bottom of the ninth to get the save.
He never even went to Yankee Stadium for the conclusion.
"We flew into New York on our way to Baltimore on an off-day," said Brett. "Since I was kicked out of the game, [manager] Dick Howser told me not to even go to the ballpark. [Airline rep] Larry [Ameche] and I went to a local bar, close to the Newark airport, watched it on TV, eating pasta and drinking wine."
Four outs later, Brett and the Royals celebrated.