Milone is the second Washington to pitcher to go deep this season, joining John Lannan, who homered on July 22 against the Dodgers.
The Cardinals' Adam Wainwright was the last pitcher to hit a home run on the first pitch he saw in the Majors, on May 24, 2006.
To top it off, Milone took a curtain call.
Before he came to the plate, Milone acknowledged that he was trying to hit the first pitch.
"I was on the on-deck circle telling myself that if I get a first-pitch fastball, I was going to swing at it," Milone said.
Once the ball landed in the bullpen, the Nats' relievers were celebrating with left-hander Sean Burnett retrieving the ball.
"The bullpen lost it. As soon as he hit it, we knew it was gone," Burnett said. "There is not a better feeling in the world for a pitcher than to hit a home run. To see the first pitch in the big leagues is something special. It's part of a special night for him. That's pretty incredible."
Milone's hitting prowess shouldn't surprise anyone, since he was 9-for-26 (.346) with three RBIs for Triple-A Syracuse.
"I take a lot of pride in my hitting," Milone said. "I used to hit in high school. I love hitting. It's fun. In my off time, if I have nothing to do, I go in the cage and hit or something like that. I love to hit. I think it's something fun other than pitching."
However, Milone started getting roughed up by the fourth, when New York scored four runs to make it a 5-4 game. Nick Evans highlighted the scoring with a two-run homer. Manager Davey Johnson took Milone out of the game with one out in the fifth after the lefty allowed a single to Jose Reyes.
Milone said he didn't lose his concentration after hitting the home run.
"I thought there was a lot more adrenaline pumping after something like that happens," he said. "I just tried to keep my composure and go out there and still pitch my game."
Despite the rough ending, Johnson ended up being impressed with the rookie left-hander, who went 4 1/3 innings, allowing four runs on six hits.
"He is a pitcher. I know everybody wanted me to stay with him," Johnson said. "He pitched well for me. He made a couple of bad pitches, but -- by and large -- he went after them. I didn't want to take a chance of him losing it."