This has happened to me many times. I usually see that if I minimize osu! for a short period of time, it will try to play catch up.
Now, playing catch up here isn't what the usual "fast forward" effect catch up. If you have cursortrail on and you minimize osu!, sometimes when you open it back up the cursortrail will appear solid. This is because there are so many frames crammed into the cursortrail and thus making it appear as a solid line.
I don't know why it does this but if left for hours on end this kind of effect might pile up. Because frankly this might not be the only thing happening, there might be other things without a visual effect that is going on in the background.
Though this post might not be helpful, I hope it gave some more info on minimizing osu!