Perkins mowed down the first two hitters with ease but then gave up a game-tying solo shot to pinch-hitter Evan Gattis to send the game to extras.
It led to Brian Duensing giving up a walk-off RBI single to Freddie Freeman with two outs in the 10th to hand the Twins a 5-4 loss to the Braves on a rainy Tuesday night that featured a 76-minute delay at Turner Field.
It stretched the Twins' losing streak to seven games -- their longest since losing 11 straight from Sept. 9-21, 2011 -- and also was their seventh straight loss in one-run games.
"It's extremely frustrating," Duensing said. "Our offense did really good tonight to battle and take the lead. But unfortunately we couldn't get the job done at the end. One-run losses are difficult. They usually make or break a season."
The left-hander served up a two-out double to Jason Heyward before intentionally walking Justin Upton to get to the left-handed hitting Freeman. But Freeman came through with the game-winning hit on a bloop single to right to complete the Braves' late comeback.
"He threw me a 3-1 slider, so I was assuming we've still got a base open at third and we had another lefty coming up, so I was just looking for another slider," Freeman said. "It wasn't the prettiest swing, but it got it done."
It came after the Braves were down to their last out in the ninth against Perkins.
"It was probably in the strike zone and on 0-1, it's a pitch I want below the zone and I didn't get it there," Perkins said. "I think he guessed right."
The late rally from the Braves happened after the Twins rallied for two in the eighth. The Twins were down 3-2 when Joe Mauer singled and Josh Willingham walked against reliever Luis Avilan to put runners on first and second with nobody out. Justin Morneau lined out softly to second, and the Braves brought in Cory Gearrin to face Trevor Plouffe.
Plouffe, who left the game after he was kneed in the head while trying to break up a double play in the 10th, came through with a single that eked through the right side to score Mauer and tie the game at 3. It set the stage for Ryan Doumit's go-ahead single to right that gave the Twins their first lead, 4-3.
The late drama came hours after Mike Pelfrey went only two innings because of the rain delay, as the Twins didn't want to take their chances after he had Tommy John surgery on his right elbow on May 1, 2012. Anthony Swarzak picked up the slack by throwing four impressive innings in relief.
"Pelfrey was lobbying pretty hard to come back in the game," Twins manager Ron Gardenhire said. "But once you get to a certain point coming off what he went through, there's no chance we're going to run him back out there."
Pelfrey had a rocky first inning, allowing two runs on an RBI double to Freeman and RBI single to Brian McCann -- both with two outs. It was a continuation of the Twins' struggles early in games, as Minnesota pitchers have allowed a Major League-worst 49 runs in the first inning in 42 games.
The Twins came back with a run in the bottom of the first on a single from Pedro Florimon off Braves starter Tim Hudson to score Willingham, who was hit by a pitch to lead off the inning.
Pelfrey got through a scoreless second, but that was it for him, as the game was delayed for one hour and 16 minutes following a leadoff single from Mauer in the third.
After the delay, Hudson remained in the game but gave up back-to-back singles to Willingham and Morneau to score Mauer and tie the game at 2.
The Twins opted to go with Swarzak for the third and he pitched well in four innings, but did serve up a go-ahead solo shot to McCann to lead off the fourth.
"Swarzak was fantastic," Gardenhire said. "It was a tough one to come up on the short end over there."