I ran #17's for a few years on my 350 with a 4bbl. No problems. I switched to #16's and saw significant improvements in power and throttle response. You will realize the most gain from your 4 bbl swap if you get 4bbl heads and a better cam to go along with it.

Adding a 4 bbl will not suck the rocker arm studs right out of the heads while you're cruising to the Dairy Queen, but if you're trying to spin the thing to 6,000 rpm you probably want to have screw in studs.