America in the 1950s and '60s remained a place where race separated people, he said on Friday during a panel discussion as part of the second Civil Rights Game, sponsored by AutoZone.
"We had to live in segregated hotels and in segregated rooming houses," Aaron said. "We still had to do the things that were not called upon as far as our white players were concerned."
Despite those racial hurdles, Aaron said he never considered whether baseball was his destiny or not.
"That never entered into my mind," he said. "I just felt like I had a job to do. I was blessed to play professional baseball, and for the players that were coming after me, I had to set an example."
Yet today, Aaron can survey the baseball landscape and not see a whole lot of others coming after him. The flood of black athletes to the Major Leagues has slowed. They have abandoned baseball or baseball has abandoned them.
That question was a central topic of discussion for Aaron and a roundtable of social activists and baseball insiders during a two-hour exchange.
The panel discussion, titled "Baseball and the Civil Rights Movement," included White Sox general manager Ken Williams; Mets general manager Omar Minaya; Martin Luther King III; Ambassador Shabazz, daughter of Malcolm X and Betty Shabazz; and Sharon Robinson, daughter of Jackie Robinson.
Their discussion was held at the National Civil Rights Museum, the site of the Lorraine Motel, where a gunman killed the Rev. Martin Luther King Jr. on April 4, 1968. The panel discussion was the prelude to a weekend of events that lead to the exhibition game Saturday between the White Sox and the Mets.
Aaron and others expressed their disappointment that the hard-fought efforts of Jackie Robinson, Larry Doby and other baseball pioneers hadn't led to the game's thriving among urban blacks.
"I'm very concerned about the decline in the number of African-American players," King said. "Somehow, we must find better ways to bring young people, particularly black Americans, into the sport of baseball."
King pointed to a troubling trend, which showed the number of blacks in baseball around nine percent. In the 1960s, the total was around 18 percent, he said.
But as King and others pointed to the problem, nobody had a solution to fix it quickly. Aaron talked about the cost of playing -- from equipment to ballfields.
Williams, who thanked Aaron for helping to open doors in baseball, said he wasn't as worried about finding more ballplayers from inner cities as he was about building better communities. He stressed the importance of better education, stronger parental influence and improved health care, which he saw as bigger priorities.
He, too, however, looked at baseball as being one component that might improve the plight of black youth.
Through all the discussion, a consensus emerged: Baseball and the civil rights movement have contributed to the betterment of America.
"It was civil rights and sports together that bring us to where we are today," Minaya said. "I don't want to sound too idealistic, but we are all in this world together."
This world included baseball, and he and Williams mentioned initiatives that Major League Baseball has undertaken to increase the participation of blacks. They mentioned the RBI program and the baseball academy in Compton, Calif., as examples.
All the participants seemed to agree that more work lay ahead if efforts to turn around the decline in black interest are to succeed.
"I really believe baseball is serious about reaching kids in urban areas," Sharon Robinson said.
Justice B. Hill is a senior writer for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.