"I got the call the next morning, congratulating me on being drafted by the Atlanta Braves. I thought, 'Lord, you didn't hear me."'
Truth is, Baker later realized, he was heard.
It was a maturing process for Baker, who was managing the Giants in 1997, when, under the direction of Commissioner Bud Selig, Robinson's No. 42 was retired across all of Major League Baseball in an unprecedented tribute. It allowed him to develop a bond with Hank Aaron, the man who assured Baker's mother that he would take care of her son. And it allowed Baker to gain a greater appreciation of what Jackie Robinson endured to help others, including Baker.
Baker heard about Robinson throughout his youth, but he didn't fully understand what Robinson faced.
Baker grew up in California, born in Riverside, and later moving north, where he attended Del Campo High School in Carmichael, outside of Sacramento.
"My last two years in high school, the only blacks in the school were my brother and me," said Baker. "The only black in the junior high school was my sister. In the elementary school, the two blacks were my other brother and sister."
Then he signed with the Braves, who not only were based in Atlanta, but had every one of their Minor League teams in the South. It was a cultural shock.
"I went from the all-white high school to an all-segregated environment," said Baker. "We were not allowed to eat in certain places [in the South]. We had to stay in black neighborhoods.
"At the same time, it was one of the best things to happen to me. I met a lot of good people in the South, white and black, which was contrary to my beliefs before I got there."
And quickly, he began to have empathy for Robinson.
"I remember things when I was a kid, inequities would come up, like rich kids making the All-Star teams because their dad's name would be on the back of the uniform as sponsors," Baker said. "I'd be an alternate. I would get mad and want to quit, and my dad would say, 'What would Jackie do?'
"I'd get in fights, and my dad would say, 'What would Jackie do?' My dad was telling us about Jackie all the time. It was always, 'What would Jackie do?' I'd tell him, 'I'm not Jackie."'
Forty-six years after signing that first baseball contract, Baker laughs at his own naïve attitude, and he is quick to acknowledge that what Jackie did opened a whole new world for Baker, who after an All-Star playing career has become one of the most successful managers in the game.
His 1,586 managerial wins are 19th on the all-time list, second among active managers to only Jim Leyland of Detroit. Everyone ahead of Baker on the all-time win list is either in the Hall of Fame or is expected to be enshrined, except Gene Mauch and Ralph Houk. Out of the 679 men who have managed in the big leagues, he is one of only 23 to have managed in at least 20 seasons.
In 10 years with San Francisco, Baker took the Giants to the postseason three times, including a World Series appearance in 2002, after which he resigned. He led the Chicago Cubs to a National League Central title during his four years at Wrigley Field, and he currently is in his sixth year with Cincinnati, which he has led to NL Central titles in two of the past three seasons.
Baker has come to realize what Jackie did.
"I have Jackie Robinson pictures on the walls in my house," said Baker. "I am teaching my son what Jackie meant. I know what he meant for me to be in this position, and how it has helped me to deal with the pressures and some of the letters I get. I was taught how to handle it by Hank Aaron, who was taught by Jackie Robinson. They were good teachers on how to be mentally and physically tough."
And most of all, Robinson pushed open doors for African-Americans in life, not merely baseball.
"He began a change in the mindset of the world," said Baker. "He didn't just create opportunities in baseball, but when baseball began to open doors, that impacted basketball and football. He was prior to the civil rights movement.
"Branch Rickey picked Jackie to be the first player for a reason. [Robinson] was well read. He was well spoken. He understood how things were and how things should be."
Robinson is praised for his restraint on the field, knowing that he had to be careful in the battles he fought. Baker, however, said Robinson wasn't a pushover.
"The Chase Park [Plaza Hotel] in St. Louis said Jackie couldn't stay there, and the Dodgers said if Jackie didn't stay, they weren't staying," said Baker. "The hotel said it was OK as long as he didn't go in the restaurant. Jackie went in the restaurant. They said it was OK if he didn't go in the pool. He dove in the pool and went swimming. They drained the pool and scrubbed it.
"He was strong enough to face the challenges and open the door."
Decades later, life is far from perfect, but thanks to Robinson, it is better.