From what I see, looks like osu will have a slower player growth (also slows the growing of toxic communities), many people don't find fun from rhythm game genres, in fast, my school is flooded with DOTA II and CS:GO players, out of x>500 peoples (grade 7 - 12), about 5 people there plays osu, but turns out I'm the only active player, with 2 players visiting a game hour a week and there is alot hype about DOTA II in my school, trading hero sets for real money or some stuffs, betting who will win and go to internet cafe together just to play DOTA II.
And since osu! isn't a "lets play" game like other players mention here...
So... New players won't likely to survive the easy difficulty (instead of choosing beginner diffs which is alot easier) and quit first.
And also because of base of everything here, because it is a japan rhythm game with half of every beatmap which is anime song realted, these will likely to ward anti-weeaboo peoples away (its true), even the song just have a anime background, some will just think the game is a hentai rhythm game.
So summary... The growth will be slow compared to many games out there due to what's inside osu, which is just average but some people out there think it negatively.