I remember posts about this very topic a few years back. I had a large amount of mouse drift on my Logitech G400(rev2). Ever since I got the G303, though, the perceived drift is so small it doesn't cause me any problems.
Here is one of the issues I discovered: I did an experiment. I bought 2 additional g400(rev2) mice to use on other computers (at work and home), 3 of the same mouse in total... And I did some rather crude mouse tests, by building a rigid frame to hold my mouse straight for X-Y axis measurements, with a slot to fit a long ruler to test measurements.
I found that every mouse, though the same model, had different move distances in all directions. The variance was between 0.5% and 2% from each other, and different for each mouse. I re-tested multiple times and the measured results were very close each time, within tolerance, so it was a consistent difference between directions.
I looked into this further, scrounging around on the internet for some tech videos on how gaming mice work, and found that this is a known feature of mouse sensors, as they are "close to perfect" as can be, but not perfect, because they rely on optical imaging for feedback (laser mice use infrared light, but it's still the same concept). The companies that make these mice also prioritise the mouse movement consistency in a single direction first, and thereafter worry about discrepancies between different directions. Their reasoning? The say the human brain can adjust for this. And it's true. It can and does. But what we are talking about is not missing notes, but your mouse getting into an uncomfortable position where you can no longer aim effectively. So far, osu! is the only game where this happens, because of the long duration between being able to re-position your mouse.
I long-believed mouse sensors didn't have a favourite direction like left, right, up or down, for how far it moved the cursor. It didn't occur to me until then. But it seems that they do, albeit very minor.
This is just one of the factors. The other factor is hand movement, and is most likely what you are experiencing, since your test doesn't produce a consistent result. We have to move the mouse in an arc to draw a straight line. The line isn't perfect either. We can't draw perfect circles either. This drift is caused by us, and requires us to compensate. Over time, drift gets less. So just practice more and it'll improve (more spinning if you must). As for inherent mouse drift that I mentioned above, there isn't much you can do except get a new mouse and hope you win the lottery (preferably one that is renowned for its sensor accuracy, because you have better chances of it being "close to perfect").
I'm not saying frame drops aren't possible, but they are almost negligible compared to these two factors.