1. osu! forums
  2. osu!
  3. Development
posted
For the past few months (since around the time of the touchscreen nerf), I have been researching and experimenting with the difficulty calculation system and potential ways to improve it.

This project was intended to be a simple effort to tweak the current system, but it has developed into something quite a bit bigger over time.

To summarise my thoughts on the difficulty system along with some other topics including pp and scoring, I wrote an essay where I discuss potential ideas to improve the systems.

Warning: If you are a closed minded person and have difficulty looking at things from a different perspective, you might as well leave now.

Please note however: This project is not completed by any means and I had planned to get a fair bit more research and experimentation done before making this public for collaboration, however I have run out of time because I am back at university now and this is my most busy semester yet.

Due to this in the essay I note various areas which require further research. Additionally, I am going to work on creating a list of issues on the GitHub repository which people are more than welcome to assist in solving.

Essay: http://puu.sh/zDbd4/4603d8fa75.pdf

Along with this essay I have made the repo with all the code from my experimentation publicly available on GitHub.

Repo: https://github.com/Syriiin/UniCircleDifficulty

TL;DR osu! is evolving and its outdated systems need to be replaced if we want to progress. Read the essay for my thoughts.
posted
  1. 1) About the changes you are proposing for the object difficulty calculation, how do you plan to calibrate the parameters in order to make them work? What criteria would you use to determine if the values are correct or not?
  2. 2) The difficulty rating of a beatmap uses the difficulty of each object and calculates a weighted sum of their values. This method is somewhat suitable for calculating the difficulty of a FC in the beatmap, but not the difficulty of non-FCs.

    I don't think having a single value for the difficulty of the beatmap is good enough, the difficulty of a beatmap should be a function of the goal to achieve (for example, you could have the difficulty of pressing all notes correctly, pressing 90% of them correctly, etc...).

    A way I think could work, is calculating a "probability of pressing each note correctly", something like:

    Difficulty: The difficulty value of the object, calculated according to the strain, angles, etc...
    Skill: The "level of skill" a player requires to have a 50% chance of pressing the note correctly.
    s: A parameter that changes the sensitivity the probability changes with changes in difficulty or player skill.

    Then, the overall difficulty of achieving certain percentage of correctly hit notes would be defined as: "The player skill that makes the probability of achieving at least the amount of correctly hit objects in the beatmap 50% (or some other fixed percentage)", in other words, the value of "player skill" that makes the cumulative density function of the associated Poisson Binomial distribution (probably approximated to a Poisson Distribution, according to Le Cam's theorem, to make calculations faster), at the amount of correctly hit notes, be equal to 0.5.
  3. 3) You suggest weighting harder objects higher than easier objects in the score calculation of a play. I don't think that it is always the case that the results on the harder notes have the most importance when determining the worth of a play. The importance of hitting a note depends on the "s" parameter of the probability function I described (the lower "s" is, the higher the weight of the object), rather than on the absolute difficulty.
posted

Full Tablet wrote:

1) About the changes you are proposing for the object difficulty calculation, how do you plan to calibrate the parameters in order to make them work? What criteria would you use to determine if the values are correct or not?
By that I assume you are asking how I would go about balancing the system? As I noted, balancing is something I haven't started on yet, but I did create a visualiser program that allows for live tweaking of values and recalculations which will help a lot in balancing. Additionally, I am attempting to gather some people from varying areas interested in helping balance it, since I'm aware it's unlikely I would be able to do this accurately alone.

Full Tablet wrote:

2) The difficulty rating of a beatmap uses the difficulty of each object and calculates a weighted sum of their values. This method is somewhat suitable for calculating the difficulty of a FC in the beatmap, but not the difficulty of non-FCs.

I don't think having a single value for the difficulty of the beatmap is good enough, the difficulty of a beatmap should be a function of the goal to achieve (for example, you could have the difficulty of pressing all notes correctly, pressing 90% of them correctly, etc...).

A way I think could work, is calculating a "probability of pressing each note correctly", something like:

Difficulty: The difficulty value of the object, calculated according to the strain, angles, etc...
Skill: The "level of skill" a player requires to have a 50% chance of pressing the note correctly.
s: A parameter that changes the sensitivity the probability changes with changes in difficulty or player skill.

Then, the overall difficulty of achieving certain percentage of correctly hit notes would be defined as: "The player skill that makes the probability of achieving at least the amount of correctly hit objects in the beatmap 50% (or some other fixed percentage)", in other words, the value of "player skill" that makes the cumulative density function of the associated Poisson Binomial distribution (probably approximated to a Poisson Distribution, according to Le Cam's theorem, to make calculations faster), at the amount of correctly hit notes, be equal to 0.5.
I think I see what you're saying, although I'm not sure I understand entirely. I kept difficulty totaling this way for now because it's much closer to the ppv2 method.

Later in the essay (in the performance and rankings section iirc) I mentioned that the single value approach had its issues and discussed the workaround of having the "difficulty value" as simply a value for show and not relevant to any actual calculations, and instead using the difficulty of individual objects to calculate performance/skill points.

This method has the issue however of storage, since the database storing difficulty values used for performance calculations would need access to hundreds or thousands of values per beatmap rather than 3.

This problem can be avoided however at the cost of accuracy (and becomes potentially exploitable) by using a regression of some kind to create a curve which values could be recreated from.

I will have to look into your idea further so understand it better, but it sounds very interesting regardless.

Full Tablet wrote:

3) You suggest weighting harder objects higher than easier objects in the score calculation of a play. I don't think that it is always the case that the results on the harder notes have the most importance when determining the worth of a play. The importance of hitting a note depends on the "s" parameter of the probability function I described (the lower "s" is, the higher the weight of the object), rather than on the absolute difficulty.
The scoring system proposed is focused at tournament play. Rewarding harder actions more points seems like the only way this makes any sense. Also isn't "s" a constant? If not, how is "s" defined for each object/difficulty point? I may be misunderstanding what you mean by that probability model though.
posted

Syrin wrote:

I think I see what you're saying, although I'm not sure I understand entirely. I kept difficulty totaling this way for now because it's much closer to the ppv2 method.

Later in the essay (in the performance and rankings section iirc) I mentioned that the single value approach had its issues and discussed the workaround of having the "difficulty value" as simply a value for show and not relevant to any actual calculations, and instead using the difficulty of individual objects to calculate performance/skill points.

This method has the issue however of storage, since the database storing difficulty values used for performance calculations would need access to hundreds or thousands of values per beatmap rather than 3.

This problem can be avoided however at the cost of accuracy (and becomes potentially exploitable) by using a regression of some kind to create a curve which values could be recreated from.

I will have to look into your idea further so understand it better, but it sounds very interesting regardless.
The difficulty calculator can calculate a difficulty value for several goals for map (for example, 100%, 99%, 95%, etc...), and then interpolate for percentages that are in-between.

Syrin wrote:

The scoring system proposed is focused at tournament play. Rewarding harder actions more points seems like the only way this makes any sense. Also isn't "s" a constant? If not, how is "s" defined for each object/difficulty point? I may be misunderstanding what you mean by that probability model though.
"s" can change for every object/difficulty point.

For example, suppose that the skill required to hit a note in the middle of a 180BPM simple stream correctly half of the time is 5, and the skill required to hit a slow, rhythmically complex, note correctly half of the time is also 5.

The "difficulty" value for both objects is 5, but they would have very different values for "s".

The difficulty of hitting a stream is mostly physical, a player that doesn't have enough finger speed would find hitting the note correctly near impossible, while if the player can easily achieve speeds faster than that, hitting the note would be rather easy, making misses rather rare. In this case, the value for "s" would be rather small.

Meanwhile, in the slow rhythmically complex note, it is expected that most players have at least some significant chance of hitting the note correctly, but that even experienced players have some significant chance of not hitting it correctly. The value for "s" is very big in this case.
posted

Full Tablet wrote:

The difficulty calculator can calculate a difficulty value for several goals for map (for example, 100%, 99%, 95%, etc...), and then interpolate for percentages that are in-between.

The difficulty of hitting a stream is mostly physical, a player that doesn't have enough finger speed would find hitting the note correctly near impossible, while if the player can easily achieve speeds faster than that, hitting the note would be rather easy, making misses rather rare. In this case, the value for "s" would be rather small.

Meanwhile, in the slow rhythmically complex note, it is expected that most players have at least some significant chance of hitting the note correctly, but that even experienced players have some significant chance of not hitting it correctly. The value for "s" is very big in this case.
Ah, I see what you mean now. I'll have to give it some thought when I have more free time, but it definitely sounds like an idea worth investigating.
posted
If pp was pushing people to improve, it would compare the score to your skill, but then a stat reset can be abused by pro players, pretending to be bad at even easy diffs, then go ham on really hard diffs.
Please sign in to reply.