The struggles…
What is the best way to get better at osu?
This question has haunted many tens of thousands of players over the years. Lots of time was spent on it in an effort to find an answer, many things have been tried and many ideas have been discarded. At some point the community agreed on certain core pieces of advice, as I present them to you here.
The 3 tenets of improvement in osu
I. Play more
II. Push your limits with hard maps
III. Play a great variety of maps and don't spam retry
These tenets seem self-evident, most people that follow them expect to benefit from them greatly, and it feels like they’ve been around forever. I suspect that one of these tenets doesn't belong on the list: The 3rd rule of improvement might turn out to be a sham.
Why do I think that? Once we're at the end of this thread, it will be clear.
Let me start off by saying that the „never retry“-rule hasn’t actually been around forever. If you go back 4-5 years and check the two most popular guides for improvement, "How to improve at osu! (by jesse1412, 400.000 views)" and "Why you aren’t improving at osu! HOW DO I GET BETTER?!?“ (by Scarletstory, 210.000 views) . Even though one of them mentions that playing a great variety of maps is benefitial, you’ll see that neither of them mention anything about how often you are supposed retry.
So where did this belief originate? I don’t know if he was the first to come up with it, but I do know that he was the first to popularize it, spreading it far and wide:
Rohulk on improvement, his ask.fm almost 2 years ago
Since then he’s been talking about it regularly on his streams, different threads made about improvement frequently include advice about not retrying, and other top players started recommending the same thing.
So... does that make sense?
Here is the logic behind Rohulk's idea as I understand it: It’s better to play many different maps because you’ll get to experience more variation. Not only do you get more consistent, but you'll also improve faster.
If you retry a lot and keep playing the same parts oft he map, you’ll improve slower and you'll be less consistent because you don’t get to live through all of the variety that you’d otherwise have.
That’s a cool theory and it does seem to make sense at face value. This theory predicts that people who retry the same maps a lot would improve way slower.
But stop for a second and look at this instead:
sourceCXu wrote:
I don't know why there's this mantra about never retrying maps. If you play a map, do something wrong, then you should also know or figure out what you did wrong, and then fix it. If you keep playing different maps, you're just letting whatever you did wrong stay in your muscle memory.
Well, now we have a real problem, what CXu said also seems to make a lot of sense! Maybe instead of playing lots of variety, we should focus on fewer maps and perfect certain patterns?
In short: we have two conflicting theories that predict different things.
- According to Rohulk's logic we should retry as little as possible and play lots of different maps to get better.
- According to CXu's logic we should exactly not do that, but instead play the same stuff again and again to fix mistakes we might be making.
Who is right and who is wrong? Are both wrong? Maybe both are right to some degree? What is going on?
And this is the problem with mere conjecture and biased self-reports: There is no real way to tell. So let's try something else instead and look at some data.
at this point I should be thanking abraker, who made this possible by writing the program that fetched the much needed data. Thanks a lot buddy! And it only took what felt like 15 minutes. You're a real beast.
After about 9 hours of running the program on the top10k in osu, we ended up with ... a crapload of data. Rank, pp, playcount, hitcount and other categories are now sitting on my computer in an excel chart that extends 10.000 rows down.
Thanks again, abraker!
So let's get started on the data. We're plotting (pp/hitcount) against (hitcount/playcount).
pp/hitcount ≈ rate of improvement
hitcount/playcount = average play length ≈ retry-rate
Explanation of the variables
Our goal is to plot a "rate of improvement" against a "retry-rate" to see if retrying really has an effect on how fast you get better at this game.
To start it off: How do we measure "improvement"?
PP and rank instantly come to mind, since those are our scales for how good someone is. Higher ranked players do tend to be better than lower ranked players. It's not true for every single pairing of players, since pp isn't a perfect measure of skill (we have people like Ming who are underranked, and DT-farmers who are probably overranked), but it generally holds up.
Next: How do we measure the rate of improvement, aka how fast someone improves? We simply view how good they are in relation to how long it took them to get there. Since "playtime" is pretty inaccurate and playcount is influenced by how much you retry, its best to use hitcount as an approximation for how much you've played this game. That's as good as it gets for now.
Given the available data, our overall rate of improvement is best described as pp/hitcount, in words: "How much pp did you gain per hit over the entirety of your osu-career". Higher values here mean that you improve faster than others.
Now we need the retry-rate. We can approximate how much someone retries by looking at the average play length. Average play length is described as hitcount/playcount, in words: "How many hits do you have per play, on average". It doesn't align perfectly with a "retry-rate", but it's at least really close. People who have a low number here are our farmers, the people who retry a map over and over after only being 150 combo in the map. And, as it will turn out, Rohulk is the single person in the top10k who gets the highest value here (by a large margin!).
You might have noticed a theme here: Our measures aren't perfect, they are approximations. Given only the data from the playerprofiles, we simply can't find a way to measure these things perfectly, the data doesn't allow it. And yet, statistics are most powerful when you have a lot of data to work with. We can hope that the sheer mass of data can compensate for that weakness (and imo, it does).
Now that we know how to approximately describe our variables, we can start plotting them against each other. Time to start with the analysis, let's see what we get!
To start it off: How do we measure "improvement"?
PP and rank instantly come to mind, since those are our scales for how good someone is. Higher ranked players do tend to be better than lower ranked players. It's not true for every single pairing of players, since pp isn't a perfect measure of skill (we have people like Ming who are underranked, and DT-farmers who are probably overranked), but it generally holds up.
Next: How do we measure the rate of improvement, aka how fast someone improves? We simply view how good they are in relation to how long it took them to get there. Since "playtime" is pretty inaccurate and playcount is influenced by how much you retry, its best to use hitcount as an approximation for how much you've played this game. That's as good as it gets for now.
Given the available data, our overall rate of improvement is best described as pp/hitcount, in words: "How much pp did you gain per hit over the entirety of your osu-career". Higher values here mean that you improve faster than others.
Now we need the retry-rate. We can approximate how much someone retries by looking at the average play length. Average play length is described as hitcount/playcount, in words: "How many hits do you have per play, on average". It doesn't align perfectly with a "retry-rate", but it's at least really close. People who have a low number here are our farmers, the people who retry a map over and over after only being 150 combo in the map. And, as it will turn out, Rohulk is the single person in the top10k who gets the highest value here (by a large margin!).
You might have noticed a theme here: Our measures aren't perfect, they are approximations. Given only the data from the playerprofiles, we simply can't find a way to measure these things perfectly, the data doesn't allow it. And yet, statistics are most powerful when you have a lot of data to work with. We can hope that the sheer mass of data can compensate for that weakness (and imo, it does).
Now that we know how to approximately describe our variables, we can start plotting them against each other. Time to start with the analysis, let's see what we get!
Every single player of the top10k is a dot on this chart
Let's explain this briefly:
- The further you are up on the chart, the faster you improve. The people that are WAY above everyone else are mostly hackers, multi-accounts, people who played offline their entire lives or touchscreen players who exploit the pp-system (the data is pre-touchscreen-nerf).
- The further to the right you are, the less you retry. The dot that is way to the right is actually Rohulk himself, so congrats to him for sticking to his principles!
The dots are clumped together pretty badly, so let's see if we can take a closer look..
enhance!
it seems to unravel after a rate of improvement >0.0015. Everyone past that point gets treated as an outlier, the people that have "something weird going on with their account".
Cookiezi is one of them, by the way. His rate as improvement is abnormally high, which isn't only the case because he's abnormally talented, but also because only his plays after the unban got tracked, meaning that his hitcount is much lower than it should be.
Let's filter out all of these players and see what we get (small note: Filtering them doesn’t actually have an effect on the trend line, since our outliers are distributed almost evenly, but it’s done here for better visibility). Here's our new chart. Rohulk is still visible on it, way to the right:
looks different! We are now looking at ~9500 points of data, ~500 were filtered out
This looks very much like low correlation.
As we can see, the bulk of the players sits solidly between 0.0002 and 0.001 ... which is quite a large range! So we already know that different players have vastly different rates of improvement across the board, the fastest players improve up to 5x as fast as the slowest players. That seems like a lot, but it can be explained rather easily. Here are just some oft he reasons:
- People are differently talented
- some people don't play for improvement at all
- some people are really good at farming and taking advantage of pp-maps wherever they can
- some people just farm S-ranks instead of challenging themselves
- some people get higher rates of improvements by playing offline
- our variables aren’t perfectly adjusted for what we’re looking for
- some players were only active in an era without pp-maps
- it gets harder to improve as you get better. Low-ranked players tend to show higher rates if improvement than high-ranked players
The trend line points down, suggesting that retrying more actually means that you improve faster! So far the data seems to support CXu's theory more than Rohulk's, if we can even allow ourselves to say such a thing because of the very low correlation.
Didn’t I just say something about low-ranked players having a higher rate of improvement than high-ranked players? Maybe we should control for that? Alright, fair enough, done:
Player #9000 to Player #10.000
No matter what group you repeat that process with, the outcome is always the same. The rate of improvement always, ALWAYS goes down as people retry less.
There are many more things we could play around with to try controlling for different things (in fact I already did, and I’m willing to discuss the results and further ideas in this thread). Take this list of effects that could mess with our data one way or another:
- We couldn't separate HR from DT players
- Players with lower average play length could show faster rates because they "try harder" at the game, while people with higher average play length play more for enjoyment (listening to the whole song, playing for the music..), people with high average play lengths don't farm as much
- Maybe the "never retry"-mantra is actually valid, but it only shows up for very high average play lengths (>400), where we have too few points of data to conduct a meaningful analysis
- "Never retry" works, but it's such a long-term strategy that most people never get to benefit because they quit before it starts paying off
- We still have no way to measure how much "variety" someone plays
As you can see, it's a complicated issue. But I want to wrap this up. This is what a large effect size looks like, one of the tenets that actually deserve to be on the list:
"Play more", large effect size, higher correlation.
In conclusion: We can’t really know if Rohulk is correct, but with all the data here I'm more inclined to believe CXu's perspective. Blindly following the theory that „sounds best“, is bad regardless. Whether or not Rohulk is correct (and he might be, as there are plenty of things we can’t control for with our data), I wouldn’t expect the effect size to be big, i.e. whether or not you retry a lot probably doesn’t matter that much. Don't treat this thread as a knowckdown-argument, it's important to remember that there are still many things that could potentially offset our observed trend.
„Play more“ still holds true, and so does „challenge yourself“, so what is there left to say?
tl;dr
It has been claimed by many that retrying a lot hurts your improvement.
After carefully looking at some data, I concluded that this claim is unfounded. The data I used are the top10k players, I plotted (pp/hitcount) against (hitcount/playcount) to approximate a rate of improvement and a retry-rate, then checked if improvement increases or decreases the more people retry.
Turns out it doesn't. More details in full version.
After carefully looking at some data, I concluded that this claim is unfounded. The data I used are the top10k players, I plotted (pp/hitcount) against (hitcount/playcount) to approximate a rate of improvement and a retry-rate, then checked if improvement increases or decreases the more people retry.
Turns out it doesn't. More details in full version.
Discuss
this post is part of a series
The reason you can (probably) never become a pro at osu
The reason why pp (probably) doesn't ruin mapping
The reason why tablet is (probably) better than mouse