"When you walk in you get an eerie feeling about what happened in there before," said Williams, who was in New York with a large gathering of Hall of Famers who were honored before the 79th All-Star Game on Tuesday.
Of the group, only one, Ryne Sandberg, had not played in Yankee Stadium. In fact, the former Cubs second baseman had never set foot there.
"I'm looking forward to it," said Sandberg. "To do it with these guys in this room, Hall of Famers, it's extra special. I'm anxious to see it. It's going to be a big evening."
While Sandberg has never visited the stadium, he certainly had seen enough games on television that were played there and he knows the two things he wants to do first and foremost.
"I just think standing there around home plate, visualizing some of the things I saw on TV growing up as a kid, like Reggie Jackson hitting a home run," Sandberg said. "I'd like to see monument park. I think just standing there and replaying all the plays and the hits and the games that I've seen on TV and see what it looks like in person."
Hank Aaron, who was baseball's all-time home run leader with 755 for more than 33 years , had a short and to-the-point answer when asked what he'll remember most.
"Winning the World Series in 1957," said Aaron, whose Milwaukee Braves dropped two of the three games played at Yankee Stadium, but still captured the crown in seven games.
Former Senators and Twins great Harmon Killebrew's most vivid memory is May 30, 1956, when Mickey Mantle hit a tremendous home run off Senators starter Pedro Ramos, a blast that struck the facade overhanging the right field upper deck, coming within feet of being the only fair ball hit out of Yankee Stadium. becoming the first ball ever to sail out of the stadium.
"A story about that home run I believe is true is that Ramos had asked Mickey before the game to sign a couple of balls for him," Killebrew said, a smile crossing his face. "And as he was running around the bases he yelled to Ramos, 'Get that ball and I'll sign it for you too.'"
Not all of the Hall of Famers have pleasant memories.
Former Dodgers manager Tommy Lasorda was on the losing end of a pair of World Series in 1977 and 1978 and will always remember the Stadium.
"I've got to forget Jackson, that's what it is," Lasorda said.
That would be Reggie Jackson, who hit three homers in a World Series game against the Dodgers in 1977 and also, depending on who you were rooting for, either was hit in the hip or stuck his hip out on purpose to get hit with Bill Russell's throw to first to try and complete a double play in Game 4 of the 1978 Series. The Dodgers thought it was interference and Lasorda argued vociferously.
"That should have been called a double play," Lasorda said, his passion still evident. "They were afraid to call it in New York and that's the way it works out."
Hall of Fame closer Dennis Eckersley pitched in the 1977 All-Star Game at the age of 22, but he remembers a game two years later.
"I gave up five home runs one night in that ballpark," he said. "I was pitching against Catfish Hunter here in 1979 and I was giving them up and he was giving them up. Four were solos, so no big deal, but the fifth one was a two-run shot. That's the memory that I have. I may have pitched some good games there, but I remember that game."
Former Orioles manager Earl Weaver brought his teams into the stadium for some epic battles against the Yankees in the mid-to-late 70s.
"Just the ballpark itself," Weaver said when asked for a memory of the place. "As a kid listening to all the World Series games, not necessarily being a Yankee fan, but hearing the announcers describe the ballpark. I don't care who you are, the first time you walk into the ballpark you're going to be impressed."
Former Expos and Mets catcher Gary Carter certainly was when he got to play an exhibition game there. It was his lone experience at the place.
"When I came to New York for that exhibition game and kind of looked around and walked out to center field and saw the monuments, I was overwhelmed because my idol growing up was Mickey Mantle. I was just in awe."
The new place will be nice, but the Hall of Famers will certainly always cherish their memories of the old place.
"They call it the 'House that Ruth Built' and some guys may have put bricks in there, but the Babe, he built it," Lasorda said. "[Lou] Gehrig, [Joe] DiMaggio, Mantle all those guys played there. What tradition, what great players."
And great memories.
Steve Gilbert is a reporter for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.