I'm assuming that you're taking too long to release on long notes or you're starting them too early.
Lets say there's 500 combo worth of long notes if you press and release on perfect judgements. If you start the long notes too early, you'll be getting more combo than a perfect score, and the same if you release them. 500 combo of long notes could turn into, say, 520 combo if you're continually off.