"I didn't know exactly what I was getting myself into," said Baylor, now the hitting coach for the Arizona Diamondbacks. "But I decided that's what I wanted to do."
The three African-American kids -- one girl and two boys -- were put in different classes at the school.
It was tough," Baylor said. "The 'N' word was prevalent. There was always resistance. "It was quite an experience growing up in that environment."
Being an outstanding athlete certainly helped Baylor in terms of acceptance, but it was still a tough assignment for someone entering his teenage years.
"The ones who participated in sports, we became closer," Baylor said. "Some of the friends that I have today, I met in seventh and eighth grade."
2012 Jackie Robinson Day coverage
When he got to Stephen F. Austin High School, Baylor was in the second class to integrate the high school and his classmates included Sharon Connally, the daughter of the Governor of Texas John Connally.
In fact, Baylor was in class with Sharon on Nov. 22, 1963, the day her father was shot while riding in the same car as President John F. Kennedy, who was fatally wounded.
"I remember the day the president was shot, the Secret Service guys came and took Sharon out of class to tell her," Baylor said. "We heard her scream."
Baylor was actually going to hear President Kennedy speak later in the day in Austin.
"School was going to be a half day and everyone was going to go downtown to see the president," Baylor said. "It was just one of those sad days walking home."
Those were challenging times for Baylor, but his baseball coach in high school, Frank Seale, helped ease the trouble. It was Seale who named Baylor captain of the baseball team.
"You always have a coach somewhere down the line that makes a difference," Baylor said. "He's been a lifetime friend. He came to the school when I was in 11th grade. We fish together, we hunt together today. I've watched his kids grow up. They didn't look at color at all."
Dealing with racism did not stop when Baylor left school. Coming up through the Minor Leagues he faced his share and even toward the end of his career, he wasn't free of it.
"I stopped a cab one night in Boston," said Baylor, who was playing for the Red Sox at the time. "And said, 'I'd like to go to the North End' and he said, 'I don't know where the North End is.' That's the United States in 1986."
One of Baylor's proudest moments came after he won the American League Most Valuable Player Award in 1979. To commemorate the achievement, the main street outside the state capitol in Austin was named after him for a day.
And while Baylor knows what it's like to be part of integration and to face racism, he knows what Robinson endured was far tougher.
"He paved the way for me as an amateur and a professional," Baylor said. "Being around Frank Robinson, Bob Gibson, Bill White, all those guys before me, I asked those guys questions all the time. Jackie was by himself. My first years in the Minors, I had a couple of guys with me, but he had nobody in the whole United States."