D-did you spend 3 and a half hours playing the same 4* song over and over, 170 times? I mean, whatever floats your boat, but I somehow doubt you are going to reach 5* in any amount of time like that.
I also came into this game with a lot of mouse accuracy from other games (RTS mainly in my case). It helps, but it's not everything. In my case, lets say that I just randomly try a song I've never seen before and try to clear it (mind you, passing is not really the best metric of success, but more on that later). If the difficulty of the song is mainly through jumps or fast slider transitions, basically things that rely only on mouse accuracy, I'll usually be fine for anything under 4.3*. However, if the song has difficult to read patterns, or (far worse) STREAMS, the limit is a lot more like 3.5-3.7*. Basically, what I am getting at is that the ability to put your mouse where you want it really fast is only half the battle. You also have to be able to keep up with what is going on, and tap at the correct time.
More importantly, I think it's generally advised that you try to achieve high accuracy on as many songs as possible within your comfort range rather than spamming one crazy hard song over and over. People talk about FCing something a lot more often than they talk about passing a song. FC, in case you aren't aware, means full combo. Basically, try to get S or A rank on as many songs as you can with FC in 3-3.5*, and gradually increase that. Of course, sometimes it's just fun to try to play some crazy fast maps, but the only thing you are going to improve by spamming one song is your muscle memory for that one song. Focus on accuracy, and play A TON of different maps (I was given the advice to never hit replay), and I guarantee that within a week passing that 4* map will seem far less challenging.
Finally, to answer your original question of how hard it really is to improve at this game, get yourself some streaming practice maps and prepare to experience true pain.