It could be many things but a quick and easy one to check is to make sure the throttle stop adjusting screw is resting on the lowest part of the step cam when at idle with the choke fully off if you are not using the stock Idle stop solenoid. I have seen this problem with some poorly made aftermarket throttle cables or poorly matched carbs and ends up holding the throttle slightly open. Engines tend to run a little slower when not fully warmed up but off the choke which would help explain why it gains a little when warm. Same goes for a vacuum leak.