"I was wondering what I was doing there," Brown said with a laugh. "You've got Willie Mays playing the outfield and Jesus Alou and Bobby Bonds there as well. It didn't seem like a good team to be with if you were an outfielder."
Luckily for Brown, who earned the nickname "Downtown" during his Minor League days with the Giants, the expansion Padres came around at the right time to jump-start Brown's career and give him a chance to play every day.
Brown, 67, makes his home outside Los Angeles in Buena Park. He still fondly remembers his time with the Padres from the expansion season of 1969 until May 1972 when he was dealt to the Oakland A's -- especially the expansion draft itself.
After all, Brown was the "original Padre," as the team used its first pick in the expansion draft on him.
"I figured that once the Giants might put me on the expansion list, thought I might have a good chance of getting picked," Brown said. "But I had no idea I would be the first one picked. It came at a good time in my career because I got the chance to play on an everyday basis."
Brown was 24 at the time of the draft and while he wasn't exactly a newcomer -- he hit 13 home runs in 446 at-bats with the Giants in 1967 -- that first season with the Padres more or less made him feel like a rookie all over again.
Brown's run-scoring double in the bottom of the sixth inning on Opening Day of 1969 drove in the go-ahead run as the Padres topped the Astros, 2-1, in front of a crowd of 23,370 at San Diego Stadium. Brown would go on to hit .264 with 20 home runs and 61 RBIs that first year and finished third in the league in outfield assists (14).
But Brown's bat was his calling card -- before and during his time with the Padres. He was named the MVP of the Pacific Coast League in 1964 playing for the Giants' top Minor League affiliate in Fresno, where he hit 40 home runs.
Maybe a legend wasn't born, but a nickname -- one that stuck with him through his final season in the Major Leagues in 1977 -- certainly was.
"I got it when I was playing in the Minor Leagues in Fresno," Brown said. "That year, I had a pretty good year. I hit a lot of balls to center field. And the way the ballpark was situated, when you did hit it over the fence, the ball was going the direction of downtown.
"One day, after I hit a home run, the radio announcer said the ball was going downtown. That's how I got my nickname."
That nickname really took off during Brown's time with the Padres, even while the team struggled to finishes of 52-110 in 1969, 63-99 in 1970 and 61-100 in 1971, as did his popularity among fans.
"The first couple of years worked out pretty well for me," said Brown, who had the best season of his career in 1970, hitting .292 with 23 home runs.
"As a team, we weren't winning many baseball games. If you're not doing that, you've got to do what you can do to improve the club."
Brown said he always appreciated the support of the fans and wishes the Padres could have had a better product on the field for them.
"It was something new to them. They came to see the Padres plus the other teams. They wanted more wins out of us. It's kind of tough to put an expansion club together with new players," Brown said. "... It helps to have good pitching. It took us a few years to come up with some decent pitching.
"At the time, we had pretty good hitting. Just didn't have enough overall good pitching."
The Padres struggled again in 1972 and Brown was shipped to the A's. He later played for the Brewers, Angels and Astros before the Phillies claimed him in June 1974. It was in Philadelphia where Brown got the opportunity to play in the postseason for the first time as the Phillies won the National League East in 1976 and 1977.
"That was fun," Brown said. "You look forward to making the playoffs and eventually going to the World Series. It was a full season of giving it your all. When you're battling to win the pennant, it makes it more enjoyable going to the ballpark."
It was in Philadelphia where Brown was a teammate of future Hall of Fame third baseman Mike Schmidt, a player that Brown says was the best he ever played with.
"I think overall, hitting for power and being able to run and play defense, it was Mike," Brown said. "He was a good all-around athlete, had good baseball instincts."
Brown retired after the 1977 season and settled into what he still calls 'semi-retired life.' Along with his wife, Sandra, they have a promotional product business. Brown said that he also has his hands full with five grandchildren, including a 10-year-old boy who has expressed interest in baseball.
If his grandson ends up playing baseball at a high level, Brown won't be surprised. After all, he comes from a family of athletes. His older brother, Willie Brown, played at USC and was drafted by the Los Angeles Rams in 1965. As a running back, Willie Brown played two seasons with the Rams and one with the Eagles before retiring in 1967.
Younger brother Oscar also attended USC and played five seasons with the Atlanta Braves from 1969-73.
The game, Ollie Brown said recently, has changed a lot from the time he played. Not so much the game itself but the players themselves.
"I don't attend many games but I still watch the game on television," Brown said. "I'm still a big fan of baseball. The game has changed. My last year was 1977 and there's been a drastic change since. The athletes now, they pretty much keep themselves in good shape year-round."