If you do not already have a good sense of rhythm, then it does take quite long to learn. That's why professional musicians start young. Even with detailed explanations of how rhythm works (it's just basic math), this is something you still have to condition yourself to over a long period of time. Every map you play teaches you rhythm. osu! makes it a bit easier than typically learning to play a musical instrument, with the same level of technical precision, since the game is essentially your rhythm trainer.
It still takes a lot of play time and you're not the first to feel like it will take far too long at your current pace. But the learning curve is not predictable. Some things take a while to learn at first, and while they may all seem like "separate" skills, inevitably an improvement in one area is linked to improvements in other areas. As a result, the rate of improvement is all over the place, which people define as "walls" (when they say they've hit a wall and can't improve) and bursts of improvement (when they are suddenly able to play faster maps). Even if you appear to be a slow starter, you can still play catch up, when things finally click into place.