Brown, nicknamed "Downtown" during his time with the Padres, was 71.
Brown played parts of four seasons with the Padres (1969-72), hitting 23 home runs with 89 RBIs in 1970.
The Padres, Brown said, gave him a new lease on a career that was going nowhere with the Giants.
"I was wondering what I was doing there," Brown told MLB.com in 2012. "You have Willie Mays playing the outfield and Jesus Alou and Bobby Bonds there as well. It didn't seem like a good team to be with if you were an outfielder."
Brown made an impact right away for the Padres.
His run-scoring double in the bottom of the sixth inning on Opening Day of 1969 drove in the go-ahead run as the Padres topped the Astros, 2-1, in front of a crowd of 23,370 at San Diego Stadium. Brown would go on to hit .264 with 20 home runs and 61 RBIs that first season and finished third in the league in outfield assists (14).
Brown said he always appreciated the support of the fans and wishes the Padres could have had a better product on the field for them.
"It was something new to them. They wanted more wins out of us. It's kind of tough to put an expansion club together with new players," Brown said. "It helps to have good pitching. It took us a few years to come up with some decent pitching.
"At the time, we had pretty good hitting. Just didn't have enough overall good pitching."
The Padres struggled in 1972 and Brown was shipped to the A's. He later played for the Brewers, Angels and Astros before the Phillies claimed him in June 1974. It was in Philadelphia where Brown got the opportunity to play in the postseason for the first time as the Phillies won the National League East in 1976 and 1977.
For his career, Brown hit .265/.324/.394, with 52 of his 102 career home runs coming with the Padres.
According to the Times, Brown is survived by his two brothers, his wife, daughter and five grandchildren.