While star rating seems to be highly correlated with the difficulty of maps, because of how the pp system works, that is not good enough for it's purposes.coldloops wrote:
Hello there,
have you tried to compare your difficulty measure with star rating ? I made a few correlation plots to illustrate this:
http://imgur.com/a/F6HjL
the "rank" is calculated by ordering the difficulty values, 1 will be the lowest, 2 the second lowest and so on.
I found it pretty interesting that it correlates with star rating so well given that they are different methods, what do you think ?
actually I made a similar analysis of beatmap diff and player skill using only score data and also got a high correlation (~0.88), so I was wondering is it really worth the effort to do this if star rating seems to be giving the same results ?
don't get me wrong, analysing score data to derive actual difficulty seems to be the best shot at getting that "true difficulty" people want but when I see those correlations I can't help but conclude that star rating seems to be pretty good already, despite not taking patterns into account.
Since the overall rating of a player puts a heavy weight on the plays that give the most pp, the error in the difficulty rating of maps that are overrated has a big influence in the overall quality of the determination of the rating of the players. In this case, the outliers in the data are more important than what correlation tests indicate.
For a person (or algorithm) to determine the rating of maps and players, the most objective way is by analyzing scores of the players. In cases where player X and player Y get a score of 700k and 800k respectively in map A, and 600k and 700k in map B; it's straightforward to infer that player Y is better than player X, and map B is harder than map A. The problem is determining how to assign uni-dimensional ratings in cases where the higher skilled players don't always get higher scores than lower skilled players; changes in the algorithm used here concern mostly how to judge those cases.
I have a new version of the algorithm (that reduces further the rating of maps where most high-skill players have low scores, but there are some low-skill players that have good scores; usually Monster, and other SV-heavy maps). I will use it for 4K maps and players (collecting scores will start in about New Year, taking scores from players in January or February, and it's estimated to finish calculating in March).