"That's all a lot of people seem to remember," Martin's son, Billy Jr., said. "The controversial stuff."
But to understand the man who passed away 20 years ago on Christmas Day, one must examine the entire picture. Martin, it seems, was many things. Above all, he was a Yankee.
Not long before signing on for what would become the first of his five managerial stints with the Yankees, Martin, then the manager in Texas, went on a Spring Training fishing trip with Rangers owner Bob Short and Yankees public relations legend Bob Fishel. Eighteen years earlier, after a much-publicized incident at New York City's Copacabana nightclub, Martin had been traded from the Yankees to the Kansas City Athletics in a move that devastated him.
His feelings for the franchise never waned, however. Fishel was a captive audience.
"And on the boat, with his boss right there, Billy was just pouring his heart out about how he'd do anything to be a Yankee again," said Marty Appel, the public relations director for Martin's first three years managing the Yankees. "So I felt very good for him when he finally did come back."
As a manager, Martin was controversial, crass and combative. He made enemies. And yet even his detractors could not deny that so much of his fire emanated from the Yankee blue logo on his chest.
"If ever there was a person who exceeded beyond his ability because of the power of that brand and wearing that uniform and wanting to hold his place with all those great Yankees stars, that was Billy Martin," Appel said. "He's probably the best example you could ever think of for what that uniform could do to somebody."
Martin never had the talent of those whose company he kept -- Mickey Mantle, Whitey Ford and other Yankee greats of his era. He was an unspectacular player. And his legacy as a manager has long been one of contention.
It is mainly for those reasons that the Hall of Fame Veterans Committee passed on Martin earlier this month when electing candidates for Cooperstown. Despite recording a comparable amount of victories and amassing a higher career winning percentage than former Royals and Cardinals manager Whitey Herzog, who was elected, Martin received fewer than three votes. Twelve were needed for entrance to the Hall.
Perhaps someday Martin will make it. Until then, other measures will have to do.
In terms of celebrity, Martin has Herzog -- along with nearly every other manager in the history of the game -- outranked. No doubt, his run in baseball was a memorable one.
Some of that stems also from what happened on Christmas in 1989. Martin and a friend, William Reedy, were traveling in upstate New York when an intoxicated Reedy drove a pickup truck off an icy road and down an embankment. According to reports at the time, neither man was wearing a seatbelt.
Martin, who was rumored to be angling for one last managing job with the Yankees, passed away and the baseball world mourned its loss.
"It was certainly the hardest thing I'd ever dealt with up until that time in my life," Martin Jr. said. "All my memories are kind of black and white from back then. I really don't have any color from what I remember."
Time has a way of distorting memories, for better and for worse. Many, and perhaps rightfully so in some cases, will always see Martin for what he was during his dark times. His son has reason to see him in a different light. Plenty give him both more and less credit than he deserves.
The one thing all of them can agree on, however -- the one thing, really, that is indisputable for a man defined by his disputes -- is that Billy Martin remains, to this day, a Yankee.
When the Yankees retired his No. 1 three years before his death, Martin told the Yankee Stadium crowd: "I may not have been the greatest Yankee to put on the uniform, but I am the proudest."
Such sentiments are perhaps what truly defined his career.
"My father was so much more than the guy that put on a good show yelling at umpires," Martin Jr. said. "He was so bright. He was such an innovative manager. If you listen to some people talk about him, some of them will tell you that without a doubt, he was the finest field manager to stand between the lines."