Contrary to popular belief, good flowing heads and a small cam will be more drivable than lesser heads and a bigger cam making the same power. It seems that you've already discovered that by the way the car feels. This is my preferred method of building an efficient, flexible and powerful engine. The newer engines coming out of Detroit are also following this formula.
Usually, a loss of driveability comes from a poorly coordinated engine package or poor tuning, not oversized ports - although that gets blamed when porting is mentioned. Most of the time it is because of an oversized cam and carb. They don't play well on the street! I see a lot of engines for sale with 500 - 1000 miles on them that have proven too much of a challenge to tune for the new owner. Usually a step or two back on the cam will fix the problem.
If the rest of your combo has the ability to flow sufficient air to make the hp and the cam is the only component holding the combo back, yes, the cam change can net you a fairly large gain in horsepower and also in torque, depending. Since you haven't listed the exhaust port flow numbers or the rest of the intake numbers, it's reasonable to assume that you haven't altered the port balance that much, and a dual pattern cam should be a better match for your combo, especially without headers.
Just did a back to back dyno session with two purpose built engines, one with parts spec'ed and provided by the customer, the other by me, for the same racing class. The first one was adequate, about on par with several other engines in the class but hampered by the cam choice. The second was a better coordinated package with the cam being noticeably better than the first one. The result: the second engine produced 12% more torque at 3500 rpm and - ready? - 45% more hp at 5800 rpm! A little more than would usually be seen!
Like Tom says, you can get a fairly big chunk of hp in the upper end of the rpm range by changing the cam to the equivalent of the RA IV cam. 450 hp should be within reach and shouldn't adversely affect your cruising enjoyment.
Fuel injection lets you go a bit farther with cam choice and still retain driveability. To illustrate, an engine that I did last year was essentially the same as the above mentioned two engines, but with more compression, bigger cam and aftermarket fuel injection for a different racing class. Produced an addition 46% hp over the best of this year's two engines but still idled at 1000 rpm and had excellent manners, very civilized. Again, don't expect this large a gain on a street engine but if you build the engine as a package you can make excellent power and still live with it.
