Yet Grant didn't let such slights affect his play, which is why his baseball career is under consideration for induction this month into the National Baseball Hall of Fame.
He joins 38 other players and executives from black baseball's yesteryear who will be considered on Feb. 27 by a panel of 12 historians and Negro League experts for induction into Cooperstown.
Had not it been for racism, Grant might have found his way into Cooperstown decades ago with some of the game's earlier black pioneers. He was considered one of the best ballplayers in the 19th century, baseball historian James A. Riley wrote in his The Biographical Encyclopedia of the Negro Baseball Leagues.
"Exceptionally quick afield and with a strong arm, he was called the 'black Dunlap' in comparison with Fred Dunlap, the best-fielding white second baseman in the 1880s," Riley wrote. "Most astute baseball observers felt he could have played in the Major Leagues if he had been provided the opportunity."
Grant, who started his career in baseball as a catcher, played professional ball for a decade. During his career, he proved a central piece on some of the top teams of the era, when black men didn't have the color barrier in place to stop them from competing alongside whites.
In 1887, Grant batted .353, hit 11 homers and stole 40 bases for the Bisons. For much of the 1890s, he starred for the Cuban Giants, and he served as the team captain for several seasons.
"He was a consistent .300 hitter with power, a fast base runner, an outstanding fielder and a popular player," Riley wrote. "He was sometimes referred to as a 'Spaniard' to make his presence more acceptable, but he still periodically encountered racial prejudice."
His play had people calling him the best all-round player for whatever teams he played on, Riley said.
Being the best wasn't enough in an era when racism reigned. Just because no legal barriers were in place didn't mean that Grant ever found acceptance, which made his achievements in the face of so much hatred all the more impressive.
In his book, Peterson quoted an article from The Sporting News that said: "About half the pitchers try their best to hit these colored players when (they are) at the bat. I know of a great many pitchers that tried to soak Grant. ...One of the International League pitchers pitched for Grant's head all the time. He never put a ball over the plate but sent them in straight and true right at Grant."
Still, Grant played on. He didn't let racism affect his love of baseball.
When the International League began to erect its color barrier after the 1888 season, Grant moved on to play in other leagues. He appeared with top black teams like the Cuban X-Giants (1898-99), the Genuine Cuban Giants (1900-01) and the Philadelphia Giants (1902-03).
Sol White, a writer who chronicled black baseball in the early part of the 20th century, called Grant "the best of his age."
"Frank Grant, in those days, was the baseball marvel," White wrote. "His playing was a revelation to his fellow teammates, as well as the spectators. In hitting he ranked with the best and his fielding bordered on the impossible. Grant was a born ballplayer."