"It was a thrill to watch the guys on TV do well and bring a championship to San Diego," he said. "That was good enough for me."
It's nearly 24 years later and Williams has been elected to the National Baseball Hall of Fame. He'll be inducted in Cooperstown, N.Y., on July 27 with Rich "Goose" Gossage, his closer on that team. While both men will go in wearing different caps -- Williams as a member of the A's and Gossage as a Yankee -- their imprint on the history of the NL's most far-flung western team is unmistakable.
Williams, in particular, was hired in 1982 after the Padres had enjoyed just one .500 season during the first 12 years of the franchise's existence. It was a good call, considering the fact that they never finished under .500 during the four years that Williams spent in San Diego, before he was replaced just prior to the start of Spring Training in 1986.
"He was a great manager, a very hard-nosed guy," said Jerry Coleman, one of the team's play-by-play broadcasters back then and still sitting in that same role today. "There was no messing around. No foolishness. What he said, you did, and if you didn't do it, you were gone."
Just like his other managerial stops, Williams stamped the club as a no-nonsense and cantankerous leader who demanded full concentration and effort from his millionaire players. In the second blush of free agency, his Padres had signed Gossage and Steve Garvey as high-priced (for the time) free agents and added Graig Nettles in a trade with the Yankees just prior to that championship season.
That 1984 group also included recent Hall of Fame inductee Tony Gwynn, playing his first full season, plus Bruch Bochy and Tim Flannery, a backup catcher and utility infielder, respectively. All three would go on to coach or manage either in the big leagues or at the collegiate level.
Williams would be an inspiration to them as they all climbed up the baseball ladder.
"He not only taught me how to play baseball, but I give him credit for teaching our fans how the game was supposed to be played," said Gwynn, now the head baseball coach at San Diego State, his alma mater, and also a Padres broadcaster. "He wasn't here that long, but his impact on our organization was a huge one. And there aren't a whole lot of people left around here to verify that."
Gwynn isn't kidding. Remaining from the Williams days in the front office is team president Dick Freeman, who left once to take a similar job with the Pirates, only to follow his bliss back to balmy San Diego.
The team, then owned lock, stock and barrel by the late Joan Kroc, is now in the hands of John Moores, who claims to know little about the Williams era.
"Dick was long before my time, but I've heard he did great things for the organization," Moores said. "He managed the team to our first pennant, what more can you ask than that? It's always good to have another member in the Hall with some Padre background."
Bochy and Flannery left in 2007 for the Giants, Bochy as the manager after 12 seasons in the same position with the Padres, and Flannery as his third-base coach in San Francisco.
Bochy, whose 1998 Padres won the club's only other NL pennant, said that Williams was a tricky individual to deal with.
"I enjoyed talking to him away from the ballpark," he said. "But when you were at the ballpark, he wouldn't converse with you a lot. He'd just let you go out there and do your job. If you didn't do your job, you were going down. But I learned a lot from him. I learned about how important fundamentals are to playing the game of baseball, doing the little things, execution. That's what Dick did for me."
Flannery -- who managed in the Padres' Minor League system, coached at the big league level and returned at the end as a broadcaster -- is particularly cut in the Williams style. Like Williams, he's a backup infielder, who spent oodles of hours on the bench and had to make the most of his limited abilities to remain active as a player, which he did for 11 seasons, all with the Padres.
Flannery, like many of his 1984 teammates in the tumultuous clubhouse of that season, didn't really appreciate Williams back then, but he certainly does now.
"He won in a lot of places and he taught a lot of people how to play the game," Flannery said. "I draw off of him every single day as a coach. I'm honored to have played for him. He's an unbelievable baseball man."
Of course, these guys all have their favorite Williams stories.
Bochy, who never played more than 63 games in a single season during his nine-year career, recalled vying for a roster spot during the final weekend of Spring Training in 1984, when Williams approached him before a game.
"Every game I'm back there, I'm catching some pitcher who struggles," Bochy said. "And he holds the catcher partly responsible. So I have to catch Mark Thurmond against the Dodgers and he asks me, 'Can you hold them under 10 runs?' That's a lot of pressure. I'm trying to make the club and here's Dick putting on more pressure."
The Padres won by a wide margin, but it didn't do much good.
"I still didn't make the club, anyway," he said.
Gwynn, who came up for good in 1983 and was pegged as a phenom, said he didn't escape the wrath of Williams, either.
"For a young player, he really forced us to think things out, because the last thing you wanted was him in your face," Gwynn recalled.
Gwynn said the team was playing at old Riverfront Stadium in Cincinnati one night in 1984 when he hit a ground ball with Alan Wiggins on first base and "kind of 75-percented it down the line." Reds second baseman Ron Oester booted it, but still threw Gwynn out at first. Williams immediately yanked him from the game, which the Padres wound up losing when Bobby Brown, his replacement in right field, committed a costly error.
"The next day, he called me into his office," Gwynn said. "He said, 'You know you cost us that game.' And I said, 'Skip, it was the third inning. You took me out in the third inning. It will never happen again.' He said, 'Yeah, but if you had run down the line, I would've kept you in the game, and Bobby Brown wouldn't have booted that ball in the sixth inning and we would've won.' It was a valuable lesson I learned that day."
Flannery said it was a long-forgotten meeting, coming just after the 1984 All-Star break that was his fondest memory of Williams' reign.
Williams wasn't famous for calling meetings, but he did on this occasion to address grousing that he was putting too much pressure on the team.
"He called us together," Flannery recalled, "and said, 'You know, it can start with a tight finger. It can go to the elbow. Sometimes it will go into the shoulder. It always connects to the back and then it goes all the way down the back. It's called a tight butt.' He shut the door and walked away. That was the only meeting we ever had. Guys were complaining he was putting too much pressure on them."
Gwynn also recalled the meeting with much mirth, bursting into laughter at just the recollection of it.
"Boy, oh boy, I have never seen grown men laugh like they laughed that day," he said. "I still use that line today. I just changed it up, but I tell the kids you can't play with a tight booty. There's nothing you can do in the game with a tight booty. You can't run, you can't pitch, you can't hit, you can't do anything."
During the Williams era in San Diego, the Padres didn't do much of anything, either, except win. And that's the essence of why you play the game.