Golson had originally been called up because the Yankees were facing two left-handed starters -- Jo-Jo Reyes and Ricky Romero -- in a three-game stretch against the Blue Jays.
Now, the trend has reversed.
"We're going to see a bunch of right-handers now, and over the last couple of days, we saw some lefties, so we made the change," Yankees manager Joe Girardi said.
Golson made an appearance in the eighth inning of Saturday's 4-1 win but didn't record an at-bat and will now head back to Triple-A Scranton/Wilkes-Barre.
Dickerson has hit .300 with a .364 on-base percentage while appearing in 31 games for the Yankees this year.
Girardi tweaked his lineup for the finale of the four-game series at Rogers Centre.
Derek Jeter had the day off, Eduardo Nunez started in his place at shortstop, Ramiro Pena was at third, Mark Teixeira started as the designated hitter and Jorge Posada got the nod at first base -- his fifth start of the season there.
Alden Gonzalez is a reporter for MLB.com. Read his columns and his blog, listen to his podcast and follow him on Twitter. This story was not subject to the approval of Major League Baseball or its clubs.