Yet he'd often tell people, "I was right on time." He'd say his life was absent regrets, and that's a belief that might well be O'Neil's endearing message to others. It should be a message that Americans, black or white, cling to.
To hold tightly to that message would be the most fitting way to remember Buck O'Neil, who died Friday at a hospital in Kansas City. He was 94.
Baseball Commissioner Bud Selig's words reflected the baseball's world's feelings.
"Major League Baseball is saddened by the passing of Buck O'Neil," Selig's statement read. "Buck was a pioneer, a legend and will be missed for as long as the game is played. I had the good fortune of spending some time with him in Cooperstown a couple of months ago and I will miss his wisdom and counsel. I have asked all clubs to observe a moment of silence before today's games."
"He is the face of the Negro Leagues in many ways," said Bob Kendrick, marketing director of the Negro Leagues Baseball Museum. "For me, personally, he was a dear friend, and I learned so much just being around him."
If any one man deserved to be the face of the too-brief history of "black baseball," O'Neil would be it.
Barred by skin color from a career in the Majors, he toiled in Negro Leagues through its glory years in the 1930s and '40s, but when Jackie Robinson and Larry Doby opened the door to the big leagues in 1947, O'Neil was too old to walk through it.
By then, he'd put away his first baseman's mitt and moved into management. He turned his attention to tutoring black ballplayers for careers in the bigs. In one sense, it might be said that O'Neil helped bury the Negro Leagues.
Viewing the history of black baseball that way might be unfair to O'Neil, whose life wasn't about destroying things but building them. One success after another led to a life in baseball as full and as rich as anyone's.
"No one has promoted baseball like he's done his whole entire life," Hall of Famer George Brett once said. "He's just been a great ambassador for the game, better than Tommy Lasorda. And he's passionate about the game."
Men with passion have no reason for regret, and O'Neil never allowed himself to have regrets anyway. He relished too much about life to bemoan any of its twists and turns.
So he never regretted the racism that denied him opportunities; he never regretted not playing an inning in the Majors; he never regretted not playing under the brightest lights; and he never regretted perhaps the biggest slight of all, not being voted for induction into the National Baseball Hall of Fame on Feb. 27, 2006.
Jane Forbes Clark, chairman of the National Baseball Hall of Fame and Museum, expressed sadness in O'Neil's passing.
"Buck O'Neil was one of the greatest ambassadors baseball has ever known. He was a giant of a man whose wisdom, kindness and generosity of spirit will live on forever in all those whom he touched and who touched him."
Still, some felt the legendary O'Neil deserved a spot in the hallowed halls in Cooperstown.
"You think of the Hall of Fame, you think of someone who transcends players and color barriers," Royals star Mike Sweeney said. "Buck O'Neil should be mentioned in that sentence."
Others shared Sweeney's sentiments. The outrage over O'Neil's exclusion proved the talk of baseball for a while. From coast to coast, people wondered aloud how the selection committee hadn't picked O'Neil.
He never shared their outrage.
"But don't weep for Buck," he told a national TV audience afterward. "Just feel happy, like I am, being thankful, like I am, that I can do and have done the things that I did do."
He did a lot in his nine decades on earth. O'Neil, who spoke at the Cooperstown induction of the 17 figures from the Negro Leagues era, built bridges between black and white, and he remained the most steadfast and most vocal ambassador of the entire scope of baseball that has lived in America.
For that alone, O'Neil deserved more than he seemed to get.
"I think he has a right to be a little bit bitter, but there's no bitterness in him," Brett said. "That's what makes him such a great ambassador because he realizes he didn't have the opportunities that some of us white people had, but there's no bitterness in him at all."
O'Neil kept himself too busy to harbor bitterness. From his boyhood to his last days on earth, he remained active in baseball. Since 1990, he had been the public face of the Negro Leagues Baseball Museum, his brick-and-mortar tribute to the men who played as well as anyone, but were kept in the shadows of the Major Leagues.
He, himself, called the museum, which he and others started on a hope and a prayer, his grandest achievement. Whatever the museum is today is a direct result of O'Neil's unflagging support of its mission -- and of baseball.
The latter was infused in O'Neil early in life. He grew up in the Deep South, working the celery fields of Florida during a period in America where Jim Crow laws held sway. For O'Neil, baseball was his escape from the hard life of a farm worker.
"A steady hitter, O'Neil won the 1946 Negro American League batting title with an average of .353 to lead the Monarchs to another pennant," wrote baseball historian James A. Riley in "The Biographical Encyclopedia of the Negro Baseball Leagues."
O'Neil played baseball in Cuba and Mexico, and he managed the Kansas City Monarchs, a Negro Leagues powerhouse, through some of its best years. He later coached and scouted for the Chicago Cubs, though he never got an opportunity to manage in the bigs.
In the 1950s and '60s, the Major Leagues weren't ready for that barrier to fall. But it did fall during O'Neil's lifetime, just as so many other barriers did in America. For some, it seems unfair that O'Neil didn't benefit personally from much of it.
O'Neil never said it, though.
In his autobiography "I Was Right On Time," he recalled a conversation he had with a writer from Sports Illustrated, who asked O'Neil about regrets.
"There is nothing greater for a human being than to get his body to react to all the things one does on a ballfield. It's as good as sex; it's as good as music. It fills you up. Waste no tears for me. I didn't come along too early -- I was right on time.
"You see, I don't have a bitter story. I truly believe I have been blessed."
So has everybody else whose life O'Neil touched.
Justice B. Hill is a senior writer for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.