Is the vaccum line connected to the modulator? You could have a leak or the modulator could be bad.
If this were happening, then every time you were to stop, and then accelerate, it would hang in fist gear unless you pulled it manually in first and then manually shifted into second etc. I think you may have to change the springs in the governor. What RPMs are you bringing it up to?