1. osu! forums
  2. osu!
  3. Gameplay & Rankings
  4. osu!mania
show more
posted
This is awesome. Talked about this in another game.
The problem is, formulating a difficulty analyzer if some sort would seem to be very difficult to make. The expounded patterns, LNs, and even SVs add up to a complex mix.
If you could give a game that has an accurate difficulty system (system/machine generated and not placed by people) i will stand corrected an also be surprised.
The 7k issue has been stated too.
Other than that there seems to be no problem with this.
posted
In my opinion, any sort of "finger placement dependent" criteria should be avoided. Many plays differently, whether it be with 2 thumbs or 2 pinkies or even 1 hand and 2 hands.

And while were at it, I doubt any kind of pattern sensitive algorithm would be able to satisfy the majority and overcome subjective difficulty. The best way to go about this, which will also be much easier to do, would be a dynamic system. A system that takes into account the performances and the general skill level of the people that plays it and uses it to determine the map's level.

To avoid data falsification we could say only players that has filled out their top plays slots, players with x-amount of plays or players which their ranking has stabilized are taken into consideration. Or really any filtering you see fit.

If no one in the top 100 can even S the map, the system will assign it an extremely high rating. If people from not even top 20k full combo it, it will receive an extremely low rating. Sure you'll get the random outliers that are better or worse at the patterns featured in the map, but the fact of the matter is the dynamic system will place this map at it's best possible place through data regression and averaging. This will overcome most of the subjective issues, the 7k/4k weighing and literally any other problems that come with deciding a difficulty for a map. The reason for this is it's based on real performance statistics and not subjective difficulty criteria.

The system was already put in place for LR2 as a third party project and really it works quite amazingly.
http://walkure.net/hakkyou/komakai.html FAQ and algorithm
http://walkure.net/hakkyou/bms.html The re-ranked charts based off clear type
posted
Considering the amount of players this should pull off well.
The stuff stated below won't be a possibility or o!m but I just want to state an example:
In the game where I came from (FtB) this was somehow practiced. In the early days most 1000 npm maps would recieve a 30/30 diff rating but as a few years passed by 1800 npm+ got the 30/30 rating and newer 1000 npms get 22-26/30 due to the increasing number of skilled players. (Patterns are taken into consideration, not just npm) Diff are votes by community so inactive voters dont get to revote.

However, in o!m, we have a very good spread of players from jhlee to n>1 star players and are also of good amount.
I don't see how this could fail. It's not like everyone will become an ET someday and that no beginners will remain.

This might also get to provide cleaner ratings (No diff will not exceed 10 stars where 10 stars is like the point of inpossibility or maybe 10.1 lol) *cough Fantazindy SHD
And yeah, since the diff will be demograhpically projected there should be no revoting problems.

All that is that needed is that new players should keep coming (not too frequently) because people improve in this kind of game. And if no new players arrive the average skill of all players increase. I've seen this happen to the point that (stating from previous example) N diffs will be like H diffs.
posted
Drace, the system would need a complete overhaul to implement something like that, and considering how the only big changes that happen are things peppy plans on doing years in advance (or at least it seems that way) I'd say there's very little chance of that happening. Changing difficulty means players can't necessarily predict how hard something will be just by looking at the number, making the entire difficulty rating system less immediately useful to players. A PP system based on changing difficulty values would also mean that everyone's PP would have to be recalculated each time the rating of a chart changed. If you got around that by finalizing the difficulty rating when a chart was ranked, that just makes the ratings even less useful for the player since the player would need to know when a map was ranked in order to know exactly what that difficulty value means. While this is less of a problem for players who keep up with the newest ranked maps, it becomes a big problem as time goes on and you get players with large collections of maps spanning multiple years.

I will admit that it would make the difficulty system much more accurate in providing actual difficulty values, but I don't think such a system will likely ever be incorporated into osu.

I agree that finger dependent pattern analysis is a no-go, because that makes assumptions about how players play, which is inherently bad when trying to make a system that judges the objective difficulty of a map.

IMO most of the 4k difficulty analysis in the first post here is essentially meaningless, because almost all of it relies on comparing specific patterns, and frankly falls under the same point I just made above: these all rely on the assumption that a player is using index and middle fingers on 2 hands separated equally, and that certain patterns are more or less difficult because of this. Sorry, but based on those assumptions alone most of your arguments on how to improve the difficulty algorithm aren't really useful.

While the difficulty system here is still not amazing, I think it's better than people give it credit for (and that's coming from someone who is really annoyed with how LNs are underrated compared to chord heavy charts).

Also, on the subject of adjusting for 5k+, it's an EXTREMELY bad idea to come up with some blanket unchanging way to adjust PP values down. That would effectively discriminate against players who focus on higher key count maps. If you were going to adjust PP because there are more or less ranked maps of a certain key count, you would need to individually adjust that based on the relative number of ranked maps of high difficulty. This would just incentivize high skill players to get scores on the difficult maps of whatever key count has the fewest hard charts ranked, which is something you should avoid.

I don't think it's a good idea to adjust the PP just because of that issue. Remember, mania still has a relatively small song collection. Be patient and a lot of these issues will eventually go away. It may take longer than we'd like, but I do think it will get better.
posted
Lots to respond to:

Drace wrote:

In my opinion, any sort of "finger placement dependent" criteria should be avoided. Many plays differently, whether it be with 2 thumbs or 2 pinkies or even 1 hand and 2 hands.

And while were at it, I doubt any kind of pattern sensitive algorithm would be able to satisfy the majority and overcome subjective difficulty. The best way to go about this, which will also be much easier to do, would be a dynamic system. A system that takes into account the performances and the general skill level of the people that plays it and uses it to determine the map's level...
The BMS difficulty scale is definitely a beautiful piece of work, but it does bring up multiple issues if you're putting it into a game with multiple keymodes that has completely different players and skillset emphases:
- If you're basing it just on the player's ability to score on the map, maps with a very short (but significant enough to affect score by quite a bit) difficulty spike will be rated significantly higher than it should be. An example would be Spy's Bangin' Burst - the bursts are very short, but the bursts affect score relatively significantly. The difficulty of the file should increase because of that difficulty spike for sure, but it shouldn't be increasing anywhere as much as a difficulty formula through how everyone performs should.
- How would ability be assigned (if you're following the formula used in the BMS difficulty system)? They have different emphases and there are multiple files that test different skills, so it would probably be difficult to quantify all these scores into one solid value that is able to encompass both 4K+ play.
- It would require regular updating for every map, which would be a huge hassle years down the road when there are tons and tons of files that require updating. Unranked maps have no difficulty number in this case as well. Even if you create a leaderboard for unranked maps, it will more than likely have an inaccurate rating because the rating scale greatly relies on the number of people playing for an accurate rating; and most if not all unranked maps will have less plays than ranked ones.

I wouldn't mind seeing the system if the first and second points is looked at and there's a way to deal with the third point, however.

Bobbias wrote:

I agree that finger dependent pattern analysis is a no-go, because that makes assumptions about how players play, which is inherently bad when trying to make a system that judges the objective difficulty of a map.

IMO most of the 4k difficulty analysis in the first post here is essentially meaningless, because almost all of it relies on comparing specific patterns, and frankly falls under the same point I just made above: these all rely on the assumption that a player is using index and middle fingers on 2 hands separated equally, and that certain patterns are more or less difficult because of this. Sorry, but based on those assumptions alone most of your arguments on how to improve the difficulty algorithm aren't really useful.
This would only be true if using different fingers would cause a huge differentiation in pattern difficulty, which isn't the case. You can use your index-middle, your middle-ring, your ring-pinky, all of these patterns will still be difficult and the differences in difficulty between them when it comes to the patterns are minor enough for them to be insignificant.

This does lead to another question though - is there an objectively best finger placement? The index and middle fingers are the most flexible and they use the same nerve. I do think that the difficulty system should be based on the most optimal finger placement, and not by any other finger placement. It's further reinforced by the fact that index-middle, middle-ring and ring-pinky styles don't have too much of a difference other than overall performance of the same patterns - they're just worse in all aspects on an objective level, if you put in the same amount of practice into all of those styles.

I don't think there are many players who use 3 fingers on one hand and 1 finger on the other enough to be significant, not to mention it's far from an optimal playing style. You have too much emphasis on one hand, runningmen on the one hand with 3 fingers would be ridiculously hard, in turn you're able to hit runningmen on the one hand with 1 finger far more easily and be able to hit either 12121212 or 34343434 (depending on which hand with 3 fingers you used), which really isn't enough to counterbalance the disadvantages for me. It's a non-optimal style of play and I don't think non-optimal style of play should necessarily be rewarded more because they're doing well on a non-optimal style of play - because by the end of the day, the player with 3 fingers on the right hand and 1 finger on the left hand will find more maps harder than the given rating than maps they find easier than the given rating. It's not really a fault in the difficulty system, but moreso a fault in the style of play.

Bobbias wrote:

Also, on the subject of adjusting for 5k+, it's an EXTREMELY bad idea to come up with some blanket unchanging way to adjust PP values down. That would effectively discriminate against players who focus on higher key count maps. If you were going to adjust PP because there are more or less ranked maps of a certain key count, you would need to individually adjust that based on the relative number of ranked maps of high difficulty. This would just incentivize high skill players to get scores on the difficult maps of whatever key count has the fewest hard charts ranked, which is something you should avoid.

I don't think it's a good idea to adjust the PP just because of that issue. Remember, mania still has a relatively small song collection. Be patient and a lot of these issues will eventually go away. It may take longer than we'd like, but I do think it will get better.
That's not what I had in mind when I'm talking about scaling PP down. I'm taking about scaling it down because there's already an inherent bias, not because of the mappool, but because of the difficulty system used. If there are more hard maps in a 7K mappool than a 4K mappool and there are two players are equally skilled at 7K and 4K respectively with the scaling implemented, the 7K player will have more PP because there are more hard 7K maps than hard 4K maps. They just won't get as much as they would without the scaling.

The point of the performance point scaling is to reduce bias, not eliminate it completely.
posted
@Bobbias

You make it sound like difficulties will be bouncing from 2 star to 8 on a daily basis :P After the initial integration the map's difficulties will stabilize to a point that their difficulties won't change more than 0.01% a day due to the large population. Sure this will in turn cause the PP values to variate, but again it'd be nothing significant enough to bounce people around in the rankings. Every ups and downs that get done is simply because that small change was needed for a more accurate system. From a user perspective, it's still top players rank top slots on hardest songs. This in no way will cause any noticeable change to the current PP and difficulty system other than boosted accuracy.

The walkure integration is completely dynamic and I've yet to see any noticeable changes despite the scale going from level -1 to 27 at 2 decimal places. Feel free to keep it open in a tab and check on it daily. You'll only see like one or two charts moving by 0.01 every week. For there to be a noticeable change you'd require the entire population to suddenly start performing better on that one song (and not the others) in just a day.

I don't really see this causing an issue with determining a map's difficulty at a glance. Everyone has their own selection of maps they like, are familiar with and play often. Just from relativity they can easily see how much harder or easier an unknown map in any sort of scaling what so ever as long as it's accurate. And this proposed system will be offering a more accurate scale, so I'd say guessing a map's difficulty is actually more difficult in the current system if we were to compare. And again, the levels wont be jumping around like some mad experiment.

Also I wouldn't regard this as a "complete" overhaul since from a technical perspective, it's a system that can easily be made in parallel with the current one. And from a user perspective, everything will work the same except the maps will be be rated more accurately. PPv2 on the other hand was a complete overhaul and it happened.

Although I didn't particularly agree with the refutes you presented, I do have to say that there are complications that would arise from this system in other regards. More specifically the fact that this could only ever work on ranked maps. Not only does this mean all unranked maps will have to use an algorithm at best, we would need to put newly ranked maps on a "difficulty rating probation" to let their difficulty stabilize. Leaving players not knowing what exactly to expect on all newly ranked charts. (Which in my opinion isn't that bad at all, playing maps with no expectations is quite fun actually. Especially since sometimes you might perform better if you didn't know that map was rated to be harder than what you can normally handle :p) But I digress. Back on point, those rather small effects such a system will have is probably enough to have people disregards it's benefits and throw it in the bin.

Finally, I agree the odds of such implementations getting made here are pretty low, but you can't exactly deem it to be impossible ^^

@Psicicite

For starters, I'm not talking about the BMS table. The BMS scale runs of community voting. The walkure algorithm I referred to is a 3rd party project "for" BMS ^^ (there's like hundreds of third party stuff for BMS zz)

What the system will do is purposed to accomplish exactly what you're worried about; overcome difficulty biases. Regardless of what a map features (keymodes, difficulty spikes, patterns) it will use the entire playerbase's performances to come with a difficulty. Any method other than raw statistics will invite biases and even errors. It doesn't matter what is featured in the map since it's regressing the actual performances of everyone that plays it relative to other maps and the player's other performances.

And also it won't require any sort of re-updating, the system is automatic. But like I mentioned to Bobbias, the system wouldn't be compatible unranked maps. Nothing's stopping us from having user assigned difficulties or just the current algorithm for unranked maps, it's what all the other games are doing lol
posted

Drace wrote:

@Psicicite

For starters, I'm not talking about the BMS table. The BMS scale runs of community voting. The walkure algorithm I referred to is a 3rd party project "for" BMS ^^ (there's like hundreds of third party stuff for BMS zz)

What the system will do is purposed to accomplish exactly what you're worried about; overcome difficulty biases. Regardless of what a map features (keymodes, difficulty spikes, patterns) it will use the entire playerbase's performances to come with a difficulty. Any method other than raw statistics will invite biases and even errors. It doesn't matter what is featured in the map since it's regressing the actual performances of everyone that plays it relative to other maps and the player's other performances.

And also it won't require any sort of re-updating, the system is automatic. But like I mentioned to Bobbias, the system wouldn't be compatible unranked maps. Nothing's stopping us from having user assigned difficulties or just the current algorithm for unranked maps, it's what all the other games are doing lol
My mistake, the walkure algorithm is a beautiful piece of work.

Anyway, that algorithm will definitely get rid of bias for sure, but the bolded part is really what I'm worried about. If you're basing it just on the players' performance (i.e. just the score), you are ignoring the fact that there are files that are harder to score on but not quite as structurally demanding. I guess the players at the lower end will struggle with the more structurally demanding map than the one that isn't but hard to score on, but I'm not quite sure how well the system would be to differentiate said difficulties. The algorithm would pretty much be perfect if it could differentiate maps like those well enough, given that it could already accurately determine difficulty of similar charts based on leaderboards perfectly.
posted
This stuff looks pretty cool. :o

Giant essay posts aside, I think it would be nice to see at least something added to mania ranking system, even if it is something small and overdue (coughcoughautoconvertscoughcough). CtB just saw a pretty sizeable change to its ranking system, maybe mania is next o.0
posted
Accorring to Tom changes aren't going to be applied very soon, but Blazier is currently more or less working on it, so changes will be there for sure. unrank autoconverts pls
posted
I don't really see the issue? If a song is actually harder on osu, it will be harder to score on osu. If a song is easy to score on, you can't just boost it's difficulty simply because there's this one part you find hard. Not only are you introducing bias, you're also rendering the whole difficulty scale obsolete by scaling it to mechanics that aren't featured in osu! and essentially giving players free PP.

I think you're thinking way too hard in this matter, the "main" purpose of the game is too score well. If it's harder to score on it, the map is simply harder. I get what you mean, under a different game following different mechanics, the said maps might indeed turn out to be harder. But were talking about osu! where we have score to represent your performance. For what purpose would you boost the difficulty rating on maps that are easy to score on over the ones that are harder? What are your standards to evaluate difficulty if not the scoring system? Game mechanics and "what ifs" that are completely irrelevant to this game?

You're saying you'd want to account for things that aren't apparent in the current scoring system within the difficulty algorithm. But I'd say it'd make much more sense to revamp the scoring system rather the difficulty scale. Or else both of them won't match with each other. The difficulty ratings should follow the scoring system, otherwise you're rendering one or the other obsolete.
posted
I just cant believe that AiAe is easier than in2006 for some people. I can almost S in2006 (need 0,5-1% more), but cant even pass AiAe :/
posted
@Drace:
Oh wow, my phrasing is just absolutely horrendous in that one. I took a while to understand how that algorithm would apply, essentially what I wanted to ask is how would the algorithm deal with scoring curves (e.g. files like Bangin' Burst, where it is really hard to 99 but very easy to S, etc.) but I phrased it in such a brief and horrendous way that it was really misleading, my mistake.

Essentially I was curious on how well that algorithm would cover scoring curves like:
- As skill goes up, the score in a file will go up in a constant rate, maybe a little anti-exponential (because the room for error goes down exponentially) - until it reaches 100% - see AiAe for example
- As skill goes up, the score in a file will go up at a decreasing rate (gradient is lower as skill increases), up to 100% - an example of a map with that curve would be Bangin' Burst [Exhaust Lv. 16]
- As skill goes up, the score in a file will go up at an increasing rate (gradient is higher as skill increases), up to 100% - I don't know any ranked maps that are like that, but most fast jumpstream maps fit that criterion.

Then I realised that the algorithm would be able to cover it well anyway, since it covers the abilities of many players in leaderboards, not just the top.

I'm liking the algorithm more and more though, and I would like seeing it as the difficulty system instead, at least for ranked maps.
posted
Ah yes indeed that's more or less of a problem that would need to be thought of if this were to get integrated.

In the BMS integration, you can see 4 different grades. You have the easy clear, groove clear, hard clear and full combo relative difficulties. (To compare, it'd about A, S, 900k+, FC in osu). You can indeed take 2 charts and realize that although one chart is easier to groove clear, the other can still be easier to hard clear.

But in osu we don't really have diffrent "clear types" and we'd be forced to regress the map's difficulty into a single level. But this isn't really an unsolvable problem, it's just require a couple personal touches to fit this environment. A couple solutions I see would be:

- We could either pick 1 point, like getting 900k, and run the algorithm off that. (lazy way out, simple integration, not accurate on every other point)

- We could pick multiple points and regress a line through them. Sort of a more advanced averaging if you may. And then pick a point on the line. (more involved, still a fairly simple integration, although not perfectly "accurate" for every point; it will be the single most "precise" result we could get for a single dimension difficulty level).

- Or finally, we could actually revamp the system to allow the use of multiple points with sort of a "target" feature. The user would be able to set a target on their song selection UI, like 900k, and the maps will accurately and precisely display how hard they are to achieve 900k score. And on top of that the PP system could be scaled to these points for an even more accurate ranking system. (relatively much more work, more of a revamp than an integration, as accurate and precise that any sort of difficulty algorithm can possibly get).

---

Also back to a previous point about finger placement dependent difficulty. Most IIDX controller players play with (wrist)thumb/middle/index/index/thumb/middle/ring. Aka, thumb/middle finger on the first 2 lanes, not everyone plays symetrically :P
posted
good ideas, i don't really have anything specific to add to this, ill let you all debate over what would work best because its hard, as kami said blazier is currently working on it but theres also this :Phttp://puu.sh/dJA2D/be8349f4d2.png
posted
Just remove autoconverts from pp system and it will be already a big upgrade in system.

Overall about pp system, things that need to be fixed is how star rating is calculating difficulty of a map, because: density = Star rating = more pp.

Like Psicicite said, star rating doesnt look on patterns but only on density of them (where its obvious that 12 34 trills are a lot easier than crossed 13 24 trills) and because of that we have maps that are over and under rated in star rating. (stamina is also a very important thing which should be considered in star rating).

And if we talking about personal experience, honestly in2006 has only one very hard pattern that makes ppl lose a lot of points, other parts of map are easier or the same hard like those AiAe quadstreams and I dont agree with that AiAe is easier than in2006 only because of 1 pattern, where AiAe requires a lot of stamina. (the same I can say that Haelequin is the hardest map because of extremely fast streams at the end which are mashable). But that its only my opinion and my personal experience from playing 4K for few years as well (just want to point out that some things can be easier or harder for other players).
posted

Drace wrote:

Ah yes indeed that's more or less of a problem that would need to be thought of if this were to get integrated.

In the BMS integration, you can see 4 different grades. You have the easy clear, groove clear, hard clear and full combo relative difficulties. (To compare, it'd about A, S, 900k+, FC in osu). You can indeed take 2 charts and realize that although one chart is easier to groove clear, the other can still be easier to hard clear.

But in osu we don't really have diffrent "clear types" and we'd be forced to regress the map's difficulty into a single level. But this isn't really an unsolvable problem, it's just require a couple personal touches to fit this environment. A couple solutions I see would be:

- We could either pick 1 point, like getting 900k, and run the algorithm off that. (lazy way out, simple integration, not accurate on every other point)

- We could pick multiple points and regress a line through them. Sort of a more advanced averaging if you may. And then pick a point on the line. (more involved, still a fairly simple integration, although not perfectly "accurate" for every point; it will be the single most "precise" result we could get for a single dimension difficulty level).


- Or finally, we could actually revamp the system to allow the use of multiple points with sort of a "target" feature. The user would be able to set a target on their song selection UI, like 900k, and the maps will accurately and precisely display how hard they are to achieve 900k score. And on top of that the PP system could be scaled to these points for an even more accurate ranking system. (relatively much more work, more of a revamp than an integration, as accurate and precise that any sort of difficulty algorithm can possibly get).
I'd definitely say that the second point is the one I had in mind. I was thinking of maybe pass / 90% (maybe this is a little too close to a pass?) / 95% / 98.5% (maybe 99%, but 99% would not cover every map and hence will give a value of infinity for that difficulty, making it impossible to regress a line - but I might be thinking of this the wrong way). Not really a fan of using score in general because it's partially combo scoring and combo scoring is kinda gross.

The third point is probably possible, but yeah it's probably a little too much work I'd say.

Tidek wrote:

And if we talking about personal experience, honestly in2006 has only one very hard pattern that makes ppl lose a lot of points, other parts of map are easier or the same hard like those AiAe quadstreams and I dont agree with that AiAe is easier than in2006 only because of 1 pattern, where AiAe requires a lot of stamina. (the same I can say that Haelequin is the hardest map because of extremely fast streams at the end which are mashable). But that its only my opinion and my personal experience from playing 4K for few years as well (just want to point out that some things can be easier or harder for other players).
It's admittedly not really fair to compare AiAe to IN2006, in my post on scoring curves, AiAe would be in the first category and IN2006 would be in the second category. You're right that IN2006 only has one really really dumb part, but the 32nd bursts are still ridiculously hard to nail (or even combo without getting 100s/worse). I can definitely agree that AiAe easily requires more stamina than IN2006, but if you're top-tier, the stamina component could be ignored quite a bit (I personally don't get very tired in AiAe) and could breeze through a good portion of the file. Even if you're top-tier, the speed/control needed for IN2006 is still very prevalent because the patterns are that difficult. I guess it's better for me to say that IN2006 is harder to score on for top-tier players, but that'd be too long-winded. Haelequin's hard section is far too short and cheatable to be considered the hardest map.

(Before anyone talks about my scores on both songs; I could get mid 97s on AiAe on average (and SDCB'd AiAe on Stepmania), but I get low 97s/high 96s on IN2006 on average, my 97.90 is a fluke)

Regardless though, with that algorithm system AiAe should be a little higher than IN2006 I think.
Please sign in to reply.