First-place Cincinnati improved its lead over the Cardinals to 1 1/2 games in the National League Central Division.
Of the 32 batters Bailey faced, 24 received first-pitch strikes. He threw 103 pitches, 77 for strikes, and had one walk with five strikeouts.
The owner of a 4-3 record -- including wins in his last three starts -- and a 3.73 ERA, Bailey has eight quality starts out of his 10 outings this season.
"He's been getting progressively better every outing," Reds manager Dusty Baker said. He had great command of his fastball. He threw some outstanding breaking balls and changeups. He minimized his pitches. He threw a great ballgame."
In eight starts lifetime against the Pirates, Bailey is 6-0 with a 1.79 ERA. His other complete game -- a four-hit shutout -- came at PNC Park on May 12, 2010.
"He was very effective," Pirates manager Clint Hurdle said. "He's been on a nice run coming in. He's gotten to a good place."
The night started out ominously as Bailey was forced to dial himself down shortly before the game when a storm system moved across downtown Pittsburgh.
"You hate to have a two-hour delay. It's tough," Bailey said. "I was just about warmed up and ready to go in the game, and all of a sudden they said we were pushing this thing back."
The delay lasted 1 hour and 54 minutes.
"The guys didn't turn it off," Baker said. "Sometimes in rain delays, it's hard to hold that intensity that you work up toward the game. The guys did a great job of settling down, and when it was time to play, they came out to play."
For the second straight game, the Reds' lineup went without a home run. But unlike Monday's loss, in which they were 0-for-7 with runners in scoring position, there was plenty of offense to go around in support of Bailey. Cincinnati was 7-for-14 with runners in scoring position on Tuesday after coming in hitting .208 in those situations.
"This was the best night we've had in a while hitting with runners in scoring position," Baker said.
Pirates starter Charlie Morton was 3-0 with a 0.93 ERA vs. the Reds last season, with two complete games. Unlike Bailey, he didn't continue a positive track record.
In the top of the second, Brandon Phillips lined a single to right field and scored on Jay Bruce's scorched double to center field. With one out, Frazier sent an RBI double to left-center field to make it a 2-0 game.
In the fourth, with one out, Frazier sent a drive through the gap in left-center field and hustled for a triple, while Bruce scored. Shortstop Josh Harrison's horrible cutoff throw to third was airmailed into the seats behind the dugout -- the error allowing Frazier to go home for a 4-0 score.
Bailey had retired a stretch of 11 in a row before back-to-back one-out singles applied some pressure. Pinch-hitter Casey McGehee rolled an RBI single into left field that scored Matt Hague. Bailey induced a Jose Tabata grounder to second base to get an inning-ending double play to escape further damage.
In the Reds' sixth, with two outs and a runner on second base, reliever Chris Resop intentionally walked catcher Devin Mesoraco to face Bailey. It backfired when Bailey notched his second single of the night, scoring Bruce.
The first six Reds batters reached in the top of the seventh on five hits against reliever Jared Hughes. Following a Joey Votto double off the right-field wall that put runners on second and third, Phillips hit an RBI single to left field. Bruce's third hit was an RBI single up the middle, and Ryan Ludwick got Phillips home with an RBI single to left field.
Bailey finished the job from there, retiring 13 of his last 14 batters -- with a throwing error by Zack Cozart being the only blemish.
"He let them hit it," Frazier said of Bailey. "In this kind of ballpark and this kind of setting, he knows his pitches are good enough to let them hit it and our defense can make the plays."
Bailey completed the game with six 1-2-3 innings.
"We were ahead in a lot of counts," Bailey said. "I thought Devin did an excellent job back there. We were pretty much on the same page all night. We threw a lot of strikes, put the ball in play and our defense made a lot of great plays. It's what it takes to go nine innings."