osu! is already targeting a niche audience with the artstyle, which is further enforced by the music that people usually map.
It's kind of regarded as a "weeaboo" game, which also means that the majority of western players are still in school, as anime is generally more popular in western societies at that age, or at least where I live. I don't really find many people that did not watch anime at a relatively young age who are interested in anime now. It's confusing to them, because not only is it in a different language, but it also has drastically different humor than one would find in western television, which is how I assume the normal family spends their time. The few exceptions are usually parents who have been introduced to anime by their children, or people who came from a different country, where the humor in the anime are quite similar to the books/graphic novels that you would find in that country.
That's why the majority of older players are actually of oriental descent, as the humor there is generally more similar to each other than to western societies, like have you seen a Chinese soap opera like what I don't even comprehend what's happening. Also, it is an assumption of mine that the majority of the players who are older usually quit after a little bit, or don't post on the forums. The reason why I believe that the majority of them quit is because they are confused by the game, or even that the game goes by too quickly for them. I've found that many adults around me find games in the first person dizzying, so they can't play them. It may be similar with osu! with the bright flashing lights and loud music. And the people that do play the game? They probably wouldn't spend their time on forums, as they are busy with other things, such as paying bills/feeding the family. Even then, with their spare time, I wouldn't think that they would go on the forums, as they would likely be confused by the immense bullshit that happens in OT or find it really awkward at least.
Have you ever gone to an anime convention? Essentially, you'd find the same people there as you would on osu!