forum

Trying to keep a clear thought on rankings

posted
Total Posts
13
Topic Starter
Sakisan
It is necessary to have rankings. With rankings people can relate to each other and have something to work for. Beating a better placed player is a big motivation at all levels. It's also the number one indicator of progress.

"With rankings people can relate to each other". This means that rankings are all about relative strength. Players that are very close in rating should get similar results on most maps. For players that are far apart in rating it should be clear which player will perform better on most of the time.

The better a ranking system, the better it can predict a player's result at any given moment. But if the only data that we have is every player's best submitted result on every map, then the rankings can only make a representation of the player's potential. Whether the potential is far off from the expected performance is another question.

The big difference between PP and TP is the kind of measurement they are based on.
TP wants to be objective at the low level. In tp all the points are translated from numbers directly recorded from the plays and the hitobjects on the map. How much points a beatmap is worth is determined by its composition and is in theory invariable. (In practice there is no perfect formula, things will always eventually change)

But 2 maps that are very similar in their basic composition could yield very different results for a lot of reasons. Maybe one of the maps has "wtf-patterns" that make you never play the map again, or the flow is incompatible to your gameplay, maybe some parts have tricky timing that put you off every time, or you don't like the song itself. Those are negative reasons, but positive reasons exist as well. Maybe the map is so popular that it's too ridiculous to even get a top 1000 on it, or you got a streak of tries where you always miss near the end making you a bit too nervous each time you get there.
It doesn't matter what the reasons actually are. While the maps are objectively just as hard, it takes more to achieve the same score on one map than on the other.

PP wants to be fair when it comes to these meta-difficulties. In pp the points are highly affected by how hard the map seems to be considering all the plays that have been done on it.

"These guys are supposed to be pretty strong but their scores are meh. You're not that pro of a player yet but you did a little bit better. Props to you for doing the extra effort, here are some extra points." The initial idea that pp would be not farmable is clearly refuted. That said, it's still an improvement to the previous score-based rankings. I like giving a little boost to people who care about their ranking. But it should remain limited.

Both systems have their merits. Anyone who denies that is keeping a closed mind or is simply provoking.

Now comes a sensitive topic.

Rankings are a social concept. The decision about who's better than someone else should really be done by a generally accepted method. If the ranking system is undermined by the community, it loses its purpose.

When I look at my country rankings, I see 2 players above me that for the time being aren't quite at my level yet. It's not only me saying it. They say it and the other players around say it too. It was socially accepted that I competed in owc instead of them. A good ranking system must be backed up by the socially accepted standards.
The 2 players are ahead of me now because of meta-difficulties that pp values. Maybe pp is right and those meta-difficulties are simply too hard for me and it's legit that their achievement outweigh my achievements. But while the community thinks otherwise, it shouldn't be the imposed as the official ranking.

But how do we assess which skills are more respected? I don't have an answer for that.
I can tell that many of you are now thinking: "speed? speed? speed!!!???"
However I'm confident that the collective consciousness of the community isn't so one sided.
I may hope that anyone working on ranking systems keep this in mind.

Do you disagree with anything I said or can you build further on this? I invite you to share your opinions in the same respectful manner that we expect the rankings to have with us.
RaneFire
Is somebody bashing you for trying to play DT with mouse-only? I think they need to rethink just how hard it is. Just because Shizuru can, doesn't mean he sets the standard for everyone.

Sadly everyone only thinks of speed. But if it's any comfort, some of us know that there is more to the game than just speed.

I think that you are right about the rankings in general, and this is why I still have hopes for PPv2. Because no system at present can properly measure the "wtf patterns" and reading complexity + overlaps. They just see 1 note at a time, or how much farming of a particular map has been going on. That said, they are pretty good and accurate at what they currently are capable of measuring.
buny
I am pretty sure everybody knows that a true ranking system is next to impossible

RaneFire wrote:

Is somebody bashing you for trying to play DT with mouse-only? I think they need to rethink just how hard it is. Just because Shizuru can, doesn't mean he sets the standard for everyone.

Sadly everyone only thinks of speed. But if it's any comfort, some of us know that there is more to the game than just speed.

I think that you are right about the rankings in general, and this is why I still have hopes for PPv2. Because no system at present can properly measure the "wtf patterns" and reading complexity + overlaps. They just see 1 note at a time, or how much farming of a particular map has been going on.
no they don't.

and even if they did, stop acting like speed isn't an aspect of the game. With speed still requires aiming and accuracy just like a slower map; the difference is, is that a map that is faster generally WILL be harder to keep up with thus resulting in harder time to aim and be accurate

Not saying that speed is the only factor in difficulty; though it is indeed a very huge factor.
RaneFire

buny wrote:

no they don't.

and even if they did, stop acting like speed isn't an aspect of the game. With speed still requires aiming and accuracy just like a slower map; the difference is, is that a map that is faster generally WILL be harder to keep up with thus resulting in harder time to aim and be accurate

Not saying that speed is the only factor in difficulty; though it is indeed a very huge factor.
I'm not acting like speed isn't a part of the game. I know it applies to all aspects of playing, not just your ZX fingers.

By "everyone" I was referring to the masses, as in those who are bad and all the others who don't use the forums. And a general statement like that really should not be taken literally, I mean come on. Of course it's not everyone! It was also in context with my previous line, not to be read stand-alone.
Topic Starter
Sakisan

RaneFire wrote:

Is somebody bashing you for trying to play DT with mouse-only?
Not at all. You do raise another interesting question. Should rankings be inffluenced by play styles ?

My aim with this topic isn't to go over specific discussions that are happening all over the forums.
How important speed is?
Mouse vs tablet?

I guess it's inevitable to bring it up. But could we keep it on an abstract level and focus on the ranking systems ?
RaneFire

Sakisan wrote:

RaneFire wrote:

Is somebody bashing you for trying to play DT with mouse-only?
Not at all. You do raise another interesting question. Should rankings be inffluenced by play styles ?
No. Although there is sometimes an advantage, players are given the option of choosing the play style that suits them best. Since they have the freedom to do so, there is no reason to influence the rankings.

Please don't take "advantage" literally. There is no advantage, I was referring to technical differences, whereas in practice, it is very different and specific to an individual's abilities and preference, and/or what he can afford to buy.
Neroh

buny wrote:

I am pretty sure everybody knows that a true ranking system is next to impossible
Yeah I agree, the variety in osu through maps and mods, spinners, accuracy etc. means it would be very difficult to give a true ranking. It's very difficult to make suggestions in how the ranking system could be improved as I'm not entirely educated on how it works.

But then again as you said:

Sakisan wrote:

Rankings are a social concept.
Rank is entirely down to how you view it, personally I don't feel I'm better than people below my rank, rather I feel I can maybe perform better in the specific maps I play a lot. It'd be very difficult to measure the specifics. I thought maybe something along the lines of separate rankings for separate performances with different mods could work. But now I think about it, after 500 players perfect a map, it wouldn't really be possible to reach the high spot much like how the ranking works in Guitar Hero. It's difficult to think at all on how it could be improved.
PlasticSmoothie
Rankings, to me, aren't necessarily a measurement of skill. It's like tennis: You need to be good to rank high, but just because you rank higher than someone else it doesn't mean you'll flat out beat them. Skill can only be measured by a human eye and a human brain, no system can take everything into account and accurately rank people according to skill.

I agree with you saying that ranking is progress. osu! doesn't have fancy new things it gives you when you've played for X hours or when you've played Y maps, the only progress we really have is skill (Because honestly, who pays attention to the levels?) and it's really difficult to measure our own skill because all we tend to see is all the people better than us and compare ourselves to them.
For example, whenever I beat a bunch of my highscores and feel like I've improved, I'll go and play 2-3 hards with mods to gain a few ranks. I'll do that if I'm really frustrated too because it makes me feel good about myself. (NO SHAME)

(Edit: @Nerohkun, your sig made my day)
Neroh

PlasticSmoothie wrote:

I agree with you saying that ranking is progress.
This pretty much describes it, ranking is just a way of showing progress. That is the idea of ranking in the first place right?

PlasticSmoothie wrote:

(Edit: @Nerohkun, your sig made my day)
I wish I knew who made it, it made my day too haha.
Tom94
Assigning difficulty to maps with a statistical approach like ppv2 is attempting to do seems to be the most reasonable to me. Difficulty directly depends on how well players perform in that approach.

The main issues I am seeing are connected to the strongly varying sample sets.
  1. Challenging mod combinations might often not be represented well enough in the total amount of plays and mods may thus have to be represented by scaling.
  2. Different people play different maps. If the better players don't even bother playing easier beatmaps, then the fail : pass ratio for instance is a lot different than if they did.
Another currently hard problem to overcome is to correctly rate scores. The currently (and arguably) only reasonable official measurement of score rating is accuracy. Reverse engineering a sensible rating from accuracy is very hard, especially because misses and 100s are so closely represented in accuracy while most people see a big difference between them. By also taking per-hitobject information about misses, 50s, 100s and 300s as well as combos into account this problem can be overcome, though. It is not _so_ unlikely for this to happen, see https://github.com/peppy/osu-api/issues/38 .
AmaiHachimitsu
Ok, I was keeping my mouth shut but since you made this topic Saki, I'll give the solution I've been thinking of since TP rank was introduced.

We all know that it's impossible to make a perfect ranking, but with some effort we can make some standards which accuracy will be very good. Enough to trust the rankings.

The solution is simple yet needs some staff and time.


Use TP's calc difficulty for maps as a base. TP is in most cases quite accurate. But if there are maps with crazy patterns or w/e element which makes TP calc not-so accurate, their worth should be changed manually. I remember that one day Airman suddenly became very pp-giving, I'm sure it was set manually.

Now, to utilise this, we need a staff consisting of people who can measure the real difficulty of the map - guys who can play and have the necessary knowledge about the game.

Tom has been nerfing some maps himself in his ranking so I don't think there is SO MUCH to do with all the ranked maps out there. Several months and it will look great.

What about newly-ranked maps? The aforementioned staff could be made to check all recently ranked maps each month. If we presuppose that 50 maps are ranked each month, I think max 5 maps would need some more in-depth checking.




Still, our minds will always be the best in assessing one's skill, but at least we can trust that top ranked player actually is a very good player
Tom94

AmaiHachimitsu wrote:

[...] Tom has been nerfing some maps himself in his ranking so I don't think there is SO MUCH to do with all the ranked maps out there. Several months and it will look great. [...]
Just to be clear: I never assigned a map a custom tp value or difficulty. Every single map goes exactly by the same algorithm. If I nerfed something, then it had an effect on all maps.

edit: i also wouldn't jump to this conclusion for pp. Airman's change in value could as well be a change in the algorithm or passing certain thresholds within the algorithm.
yoyomster

Sakisan wrote:

The big difference between PP and TP is the kind of measurement they are based on.
TP wants to be objective at the low level. In tp all the points are translated from numbers directly recorded from the plays and the hitobjects on the map. How much points a beatmap is worth is determined by its composition and is in theory invariable. (In practice there is no perfect formula, things will always eventually change)

But 2 maps that are very similar in their basic composition could yield very different results for a lot of reasons. Maybe one of the maps has "wtf-patterns" that make you never play the map again, or the flow is incompatible to your gameplay, maybe some parts have tricky timing that put you off every time, or you don't like the song itself. Those are negative reasons, but positive reasons exist as well. Maybe the map is so popular that it's too ridiculous to even get a top 1000 on it, or you got a streak of tries where you always miss near the end making you a bit too nervous each time you get there.
It doesn't matter what the reasons actually are. While the maps are objectively just as hard, it takes more to achieve the same score on one map than on the other.

PP wants to be fair when it comes to these meta-difficulties. In pp the points are highly affected by how hard the map seems to be considering all the plays that have been done on it.
I think an accurate ranking system should use objectivity component of the tp system as a basis and fairness component of the pp system for fine-tuning. In this way both sytems complement each other. Each system has its merits, but it also has its flaws. Each system's merits can make up for the other one's flaws.
Please sign in to reply.

New reply