Guest
home
beatmaps
rankings
community
store
help
forum
  1. Forums
  2. osu!
  3. Development

QAT Restructure Follow-Up

posted
Total Posts
105
Mao
Elite Mapper II
2,963 posts
Joined December 2012
Topic Starter
Mao
Hello everyone! After some discussions about the QAT rework thread, we are happy to present you our proposal for a new system.

Proposal [Updated] | Proposal


Please note that this is still a proposal and that we are taking feedback. If you want changes to be made to it, please post in this thread until Monday, February 4th 23:59 UTC+0. After that, we will try to incorporate your feedback as well as we can!
Last edited by Mao , edited 1 time in total.
UndeadCapulet
1,100 posts
Joined April 2013
UndeadCapulet
kowai
Megapcmx
289 posts
Joined May 2014
Megapcmx
uh
Feerum
osu! Alumni
2,042 posts
Joined August 2014
Feerum
Sounds interesting, but i have a little problem with this part:

Applications for joining the Beatmap Nominators will mostly stay the same but instead of holding them every three months, they will be open at any point in time.
The applications will still be composed of two parts, a ranking criteria proficiency test and a manual evaluation by members of the “Management Team”.

If you are removed from the Beatmap Nominators, you will be put on a cooldown of three months before you can re-apply. If you leave the Beatmap Nominators on your own merit though, you will be allowed to re-join within the next two weeks after your removal.


Okay so, sounds not bad but there is one aspect not explained. What if someone applied and got rejected. Will this person be set on a cooldown before they can re-apply or not?
It's a pretty important part. If there is no cooldown, people who got rejected and "Do not accept the decision" or pretty stubborn persons would spam their applications all the time and the "Management Team" would have to re-evaluate this person all the time new.
Chanci
253 posts
Joined July 2017
Chanci
this is v nice i am pogchampioning out of my seat
Hydria
Beatmap Nominator
1,245 posts
Joined April 2011
Hydria
Give the Disqualify button to the Beatmap Nominators
The Beatmap Nominators will take on the current QAT’s job of acting on reports in the Report a Qualified beatmap here! thread.

Vetoes will be decided by the Beatmap Nominators
Instead of a small group deciding about the fate of a map, the “Management Team” will initiate a majority vote among all BNs (except for the one who placed the veto) of the respective mode.


Top 10 plans to fail within weeks, guaranteed barely anyone will be trying to dq controversial or even just subjectively bad maps by april because it won't be worth the time and effort to do so.
Nao Tomori
osu! Alumni
3,106 posts
Joined December 2014
Nao Tomori
You should clarify "If you are removed from the Beatmap Nominators, you will be put on a cooldown of three months before you can re-apply." to include "or your application is rejected" since that's presumably how the system would work.

I am concerned about 2 things. first: majority vote for all vetoes. I think this would be a lot of overhead and think that randomly selected pool of x amount of BNs would be better to avoid having to ping every BN and get their opinion on vetoed maps over and over, which would get tiring. Think of it like jury duty or something. If people are always asked "is this map which you really don't give a shit about rankable" then it will lose a sense of meaning resulting in people not looking at the maps or vetoes in detail and just saying yeah it's fine whatever.

second, and more importantly: unrankables and objective issues are not actually important or a good representation of a BN's skill. for example: BN A consistently pushes maps which have complex timing and snappings, which occasionally get DQ'd for some adjustments to timing or snapping. BN B exclusively nominates single bpm NHI 1/2 based anime maps which never get DQ'd.
BN A is at much, much, MUCH higher risk of getting kicked than BN B despite nominating more diverse or interesting content than the usual fare, and neither of the two maps getting DQ'd for being bad maps. Only counting objective DQ's does not mean that people can nominate risky maps - it just changes the definition of risky from "uses weird techniques" to "has an unquantized mp3" and doesn't address the core issue of people only wanting to nominate single bpm 1/2 based anime maps to avoid getting kicked for timing DQ's.
This is a fundamental problem with how QATs gave up on caring about map quality and only focusing on unrankables - unrankable issues are not major problems for the most part. A wrongly snapped slider takes a grand total of 2 seconds to fix. A 10ms red line adjustment is not a major problem that causes a good map to become terrible. Meanwhile people nominate all forms of complete and utter trash (Uta intro ver anyone?) which don't have unrankables and therefore suddenly they are fine.
I'm not saying to magically start giving a shit about mapping quality but this idea that a BN getting DQ's for timing is worse than a BN not getting DQ's because they nominate rankable trash is really stupid and shouldn't be in the score system. The score system should just be activity and SERIOUS unrankable issues (entire sections missing hitsoundings, large amounts of unsnaps due to fucked up green lines, 20+ms offset issues, etc.)
Last edited by Nao Tomori , edited 1 time in total.
Kroytz
703 posts
Joined February 2013
Kroytz
"Disband the Quality Assurance Team"
All I needed to read.
Annabel
Beatmap Nominator
565 posts
Joined October 2013
Annabel
The “Quality Assurance Helpers” sub-group inside the BNs will continue checking every qualified map for unrankables and objective issues until an automated method replaces them. Moreover, the QAH will be renamed to avoid confusion with the former QAT.

The Beatmap Nominator rules will be updated so that a map may only be disqualified if unrankable or objective issues are present, the mapper requests it, or in the event of a veto.


Does this mean that probation BNs will also be able to DQ maps?

Beatmap Nominator Applications will always be open
Note that implementation of this system might take some time to prepare

Applications for joining the Beatmap Nominators will mostly stay the same but instead of holding them every three months, they will be open at any point in time.


Having the applications always open means that there would always be probation BNs, so would probation just disappear? Because I don't know how it could be done to be constantly looking over new people that don't come in a cycle or any set number, just random.

Just sounds a little confusing right now. ):
Last edited by Annabel , edited 1 time in total.
FrenZ
304 posts
Joined January 2017
FrenZ
You guys should add a cooldown between how long people can apply for BN - perhaps two weeks or something. People could just spam requests every other day under this proposal since it's not listed...

edit: Hi Feerum! I didn't reload my page until after I posted this and you mentioned this already.
Last edited by FrenZ , edited 2 times in total.
Skidooskei
377 posts
Joined April 2017
Skidooskei
Maybe make the name something more based towards mapping? "Management Team" seems to general and reminds me of some office job lmao.

Even "Mapping Management Team" would be better imo.
Last edited by Skidooskei , edited 1 time in total.
Izzywing
osu! Alumni
2,251 posts
Joined March 2016
Izzywing
Main thing I took away is that it’s just the current system but everything’s appropriately labeled instead
Ascendance
Elite Mapper
4,252 posts
Joined July 2013
Ascendance
T_T
Deca
152 posts
Joined October 2016
Deca
Re:vetos and votes

Will every BN be required to vote?
Are BNs allowed to abstain from the vote?
If yes, and a large majority of BNs opt to abstain from the vote (so that a very small portion of BNs actually voted on the map), what will happen?
Will probation BNs vote on vetos and if so, will their votes be as evenly weighted as full BNs?
Monstrata
Elite Mapper: Aspirant
4,642 posts
Joined May 2013
Monstrata
Main Issues:

Majority Vote regarding Vetoes

It isn't feasible to hold a vote across the entire BNG. Something like selecting x amount of readily available BN's would be better. I think 7 is the most feasible. Selection should be random, of course, and participation in vetoes should become a mandatory responsibility of BN's. Currently, it seems as if people who don't want to participate in votes can just not. There is no penalty for not participating, but as a result the vote may be skewed. Therefore, in order to ensure participation, I feel BN's who are chosen to participate in any veto mediation vote must also provide a short opinion on their choice (yes or no). I think it only has to be 2-3 sentences, nothing major, but this is to at least acknowledge that the BN has participated in the veto and isn't just going to blindly vote yes/no to everything and be non-toxic. (Voting no to a veto mediation is NOT being toxic, let's not set any bad precedents).

What happens after a Veto?

This one's rather straightforward. After a veto has been placed, and has been maintained through voting, the mapper is then forced to change the pattern or leave the map to be graved. Who will lift the veto? I'm assuming the original BN. But what if they continue to maintain that the mapper hasn't fully addressed the issue, or the mapper continues to call the veto'ing BN back to recheck without completely fixing the issue?

I want to add two additional rules to flesh out veto mediation scenarios:

- After a veto has been maintained, If the veto'ing BN does not recheck (and/or approve) the changes made to the map one month after the mapper has called them back for a recheck, the veto will be lifted. This will resolve both issues. It allows for mappers to "escape" a veto if the BN is not willing, or has grown tired of enforcing the veto and constantly rechecking the map. However, it also allows the BN time before having to recheck. If the mapper continues to call the BN back without addressing changes adequately, the BN is also welcome to take other requests, ask the mapper to rethink their concepts, and recheck a few weeks later. I think this is fair, as some vetoes for core issues are not fixable without major reworks anyways.

- If the veto'ing BN is removed, any of the BN's who voted "no" in the veto mediation can replace the BN and uphold the veto. If no one wishes to do so, then after 1 month, the veto will be lifted.

How to get into Management Team?

I'm sure a few people (hi Ascendance) will be interested primarily in this. Is there a process to get into Management Team? And what will happen to other aspects of the mapping/modding community? I'm talking about pushing forward new amendments and changes to the Ranking Criteria mainly. (Which is another subset issue). How will people be chosen for this specialized team?

BN Acceptance

Will there be any change to the way BN's are accepted going forward, now that reviews are no longer every 3 months but on a rolling basis? Will activity count be lowered etc...? I'm also interested in changes to selection criteria. Seeing as how this can potentially mean a lot more BN's entering more frequently, (as well as exiting).

Also if someone is rejected, what is the cooldown before they can reapply again?

Another note: I feel severity of a kick should be considered too. For example, BN's kicked for missing too many unrankable elements, or are inactive etc... should not be subject to a 3 month cooldown imo. I feel that for unrankable elements, this suggests the BN needs more training, or isn't thorough enough, but 1-2 months should be enough. For inactivity, well, can't a BN just resign if they know they will get kicked for inactivity? Then they would be able to reapply 2 weeks later. On the other end, I think behavior-related kicks could be 3 months. I think having a flat # of months is not necessary, and you guys could change the cooldown to suit the specific case of the BN being removed. Basically, make cooldowns more case-by-case and not necessarily do 3 months for everyone.
Arzenvald
4,058 posts
Joined July 2013
Arzenvald
Another change ugh..
I'll just follow along, dunno what to say..
Drum-Hitnormal
Beatmap Nominator
1,727 posts
Joined March 2011
Drum-Hitnormal
why is BN score based on issues rather than # of maps ranked?
pimp
1,417 posts
Joined April 2010
pimp
i don't like the idea of giving the DQ button to all nominators but everything else seems okay i guess.
Aurele
Global Moderator
7,514 posts
Joined January 2011
Aurele
I like where this is going at the moment.

However, I am having some concerns.

New Beatmap Nominators // Probation
How are new Beatmap Nominators and Probations going to be handled? Will they be able to participate in disqualifications, vetoes and majority votes?

Application cooldown
- Is there any cooldown for potential Beatmap Nominators to re-apply after they fail the test? If so, how long would it be?
- Just a clarifications, is there a cooldown for nominators leaving by their own? The follow sentence is a little confusing to me.

"If you leave the Beatmap Nominators on your own merit though, you will be allowed to re-join within the next two weeks after your removal."

Does it means that a nominator leaving on their own will only have two weeks to come back to the team? If so, this short period of time seems unnecessary. In this case, it would only require the nominator to send in a "Leave of Absence" for said time. Instead, give the chance to the nominator to join back in before the next 30 days after their removal.
Or does it means they can join back 2 weeks after their removal?
Refills
318 posts
Joined October 2017
Refills

dudehacker wrote:

why is BN score based on issues rather than # of maps ranked?

beatmap nominators are in charge of, in essence, deciding what maps get ranked. therefore, under this, a bn doesn't actually need to have a ranked map, they just need to know what is high enough quality for ranked, and be able to spot issues within a map

tl;dr - a bn doesn't need to be a good mapper, but they do need to be a good modder
squirrelpascals
984 posts
Joined March 2015
squirrelpascals

Proposal wrote:

The Beatmap Nominator rules will be updated so that a map may only be disqualified if unrankable or objective issues are present, the mapper requests it, or in the event of a veto.


So does this mean that "dqs for discussion" are essentially dead? This means that if the community is still discussing a controversial map that gets ranked in one day, nothing is stopping it from getting ranked. Unless a bn initiates a "veto for discussion?" Which sounds like a complete work around and totally separate from a veto's true purpose. Controversial maps are controversial for a reason, so I think that room for discussion amongst the community is important so that we can reference to it for similar future maps.

Proposal wrote:

A new score system for Beatmap Nominators will be introduced
However, instead of focusing on subjective issues, this system will only account for unrankable and objective issues.


It doesn't sound like it makes any sense to score bns only on unrankable issues they find, since that doesn't really consider the initial quality of the maps modded, which can give a false perception of a bn's activity.

Say, for example, bn #1 checks 20 maps that are all from fairly experienced mappers who know what they're doing, so he finds about 5 unrankable issues in total. bn #2 likes to help newer mappers out who are becoming more capable of ranking stuff, so he checks 4 maps in total and finds 10 unrankable issues. Assuming all the unrankable issues of all maps are found, bn #1 is pushing 5x more maps than bn #2 is through the ranking process. But under this scoring system bn #2's score is still 2x higher.

I've tracked all the unrankable issues I've found under my modding spreadsheet which needs to be updated, and using data from the past 5 months I found that the average amount of unrankable issues I find in each map comes to ~2.5. So I think a formula we can start from, measuring the amount of unrankable issues found per map, can look something like

(#total unrankables / #maps modded) / 2.5

So a score of 1 means you found an average amount of unrankables per map. Maps with no unrankable issues (from more experienced mappers) and maps that have a lot of unrankable issues (from newer mappers) will pretty much cancel eachother out.


I wrote a lot but I get this is still being developed :p Besides for those 2 things this proposal sounds super dope!
Last edited by squirrelpascals , edited 4 times in total.
Nao Tomori
osu! Alumni
3,106 posts
Joined December 2014
Nao Tomori
@dudehacker they never want to rank bns based on how active they are because then all the inactive people start yelling about how the active ones just yolo rank everything instead of thinking about quality =D
also they will never reward bns, so having a system that is positive (# of maps iconed) makes no sense because the only thing bns will ever get is punished; a negative system (# of maps dq'd/popped) fits that much better
Kawawa
Kawawa
＊Give the Disqualify button to the Beatmap Nominators
＊Vetoes will be decided by the Beatmap Nominators
I think that even though BNs will have the power to veto maps, I am especially worried that mini-game BNs will not use this power as often as they should due to either disinterest or a fear of backlash. BNs will be in the same situation as QATs before them where they are afraid to do a veto or afraid to vote yes on veto votes for subjective issues. there is also a chance that circlejerk vetoes will happen.

The QAH role is not as clear as it should be either since their main job was to assist the QAT. If the QAH has to directly veto instead of report less people will want to join due to having much more pressure on them than the average BN. There isn't really any reward for BNs doing a good job with this new system, instead it punishes them with more opportunities for community criticism and higher chance of kicking imo.
so not sure that QAT must be removed. I'm still think the current system is pretty solid, since it is just a inside problem.
Last edited by Kawawa , edited 1 time in total.
Mao
Elite Mapper II
2,963 posts
Joined December 2012
Topic Starter
Mao

Monstrata wrote:

It isn't feasible to hold a vote across the entire BNG. Something like selecting x amount of readily available BN's would be better. I think 7 is the most feasible. Selection should be random, of course, and participation in vetoes should become a mandatory responsibility of BN's. Currently, it seems as if people who don't want to participate in votes can just not. There is no penalty for not participating, but as a result the vote may be skewed. Therefore, in order to ensure participation, I feel BN's who are chosen to participate in any veto mediation vote must also provide a short opinion on their choice (yes or no). I think it only has to be 2-3 sentences, nothing major, but this is to at least acknowledge that the BN has participated in the veto and isn't just going to blindly vote yes/no to everything and be non-toxic. (Voting no to a veto mediation is NOT being toxic, let's not set any bad precedents).


Randomly selecting BNs was my first thought too though but in that case, we'd need to anonymise it as a fewer number of people is probably more prone to bribery. Also in that case the mapper must have a chance to contest the decision made by the BNs because if you are unlucky and get 7 people selected that all are in favour of the veto despite maybe most of the other BNs being against the veto, the decision would not be representative of the whole group.

Monstrata wrote:

How to get into Management Team?

I'm sure a few people (hi Ascendance) will be interested primarily in this. Is there a process to get into Management Team? And what will happen to other aspects of the mapping/modding community? I'm talking about pushing forward new amendments and changes to the Ranking Criteria mainly. (Which is another subset issue). How will people be chosen for this specialized team?


This has yet to be decided but I think it will probably go into the same direction as QAT promotions in 2017 pre-upheaval where when somebody shows good management skills (e.g. managing different projects well) and good behaviour (as they will most likely have the same permissions as QAT have right now) they will be brought up in the "Management Team" and a majority vote/discussion will happen.

Monstrata wrote:

Also if someone is rejected, what is the cooldown before they can reapply again?


Yeah, this has been an oversight and totally needs to be added. I think 3 months would work well as that's the current length of a cycle but I personally wouldn't mind reducing it either.

Monstrata wrote:

Another note: I feel severity of a kick should be considered too. For example, BN's kicked for missing too many unrankable elements, or are inactive etc... should not be subject to a 3 month cooldown imo. I feel that for unrankable elements, this suggests the BN needs more training, or isn't thorough enough, but 1-2 months should be enough. For inactivity, well, can't a BN just resign if they know they will get kicked for inactivity? Then they would be able to reapply 2 weeks later. On the other end, I think behavior-related kicks could be 3 months. I think having a flat # of months is not necessary, and you guys could change the cooldown to suit the specific case of the BN being removed. Basically, make cooldowns more case-by-case and not necessarily do 3 months for everyone.


The numbers in the proposal are still quite arbitrary and will probably be adjusted when we polish it. I like this idea though!

pimp wrote:

i don't like the idea of giving the DQ button to all nominators


What's the problem with that? The BN rules will be adjusted to forbid any kind of abuse and I believe we can trust BNs with another button, especially since it can't really break anything.

______

We also still need to figure out how to handle probation exactly. I have seen very strong opinions on this from both sides - while some people think it's very important to see if new guys are fit for the role and to give people who messed up another chance, others think it should be replaced by a system like the old three strike system where whenever you have a major messup, you get stricken and once you accumulated three strikes you get removed.
I'm not entirely sure which one of these works best with the proposed systems so I'd like to get some more thoughts on that here too.

Moreover, I wouldn't get too hung up on the score system yet as its direction heavily depends on implementation. I feel like the best way to go about it would be a system where both the amount of dqs, their severity and the BNs activity are calculated into a score and then once a BN drops below a certain threshold, a bot pings the "Management Team" and they'll look closer into that BN's case.

We will also try to keep you updated with the score system when concrete planning has started, i.e. there will most likely be a separate proposal for that in the future.
Last edited by Mao , edited 1 time in total.
Kibbleru
osu! Alumni
5,846 posts
Joined August 2013
Kibbleru

FrenZ396 wrote:

You guys should add a cooldown between how long people can apply for BN - perhaps two weeks or something. People could just spam requests every other day under this proposal since it's not listed...

edit: Hi Feerum! I didn't reload my page until after I posted this and you mentioned this already.


I believe that was the plan tbh. Should be 3 month cd after getting rejected pretty much.

Anyways, liking this proposal better than the other one
Last edited by Kibbleru , edited 1 time in total.
celerih
617 posts
Joined July 2014
celerih
Very much for this proposal. It's basically taking the current system and making it match the reality of things, which is pretty nice to do.

My only gripes are basically entirely covered by nao in this post, especially that first point. That second point would be nice definitely, but u get the same issues with that as you get with QAT enforcing quality standards onto maps, so I don't expect anything to be done about that
Mafumafu
Nomination Assessment Team
3,372 posts
Joined August 2013
Mafumafu
Second this.

Actually I would suggest that the Management Team should also take responsibilities or initiatives regarding other mapping/modding-wise events to revive them, like Spotlight, Contest Organization, RC Amendment, Education and Mentorship etc. I believe some of these are in a stasis and/or require more hands. It will make the main tasks of the team to be more higher-level, and genuinely "management and leadership-wise". These events would actually, in my eyes, make the team do greater for the entire community than dq'ing over mapsets xd.
lit120
3,272 posts
Joined August 2013
lit120
Are we even back to BAT's era now?
Fu Xuan
Beatmap Nominator
1,433 posts
Joined September 2015
Fu Xuan
Pog
squirrelpascals
984 posts
Joined March 2015
squirrelpascals

Mao wrote:

We also still need to figure out how to handle probation exactly.


For starters, I'd like to suggest telling the bns on probation generally what Management will be looking for when they're evaluated. My readmission felt a bit like the QAT was saying "ok you're a bn now but you're also on probation, so yeah gl." There wasn't really much guidance as to how anyone on probation was being evaluated. Were the QAT looking for number of icons? bubble pops or dqs? activity? I got a lot of mixed responses and some rumors from people I asked (bns and like 2 qat i think). For myself I had the reasons from my previous probation failure to focus on improving but I didn't see any clarity as to how our performance was being measured for the other probation bns. This just put pressure on me to be perfect in every aspect of modding, which I know isn't the true expectation of the QAT.

This whole ordeal made probation super super stressful for me, and it made it a bit of a mystery as to why some of the people didn't pass. Probation doesn't need to be stressful or mysterious. We need to write some sort of probation quidelines or something.
Last edited by squirrelpascals , edited 1 time in total.
Fycho
Global Moderator
2,486 posts
Joined August 2012
Fycho
A team should solely focus on doing something so that the team can keep maintained. I like the idea that MT(Management Team) "only" focuses on something team-wisely, as we don't want to see there are different branches across the MT like the branch QAT in 2017. Stuffs like Education or Mentorship shouldn't be the responsibilities of the MT. They can also be done by members of MT individually, and these stuffs are mostly community-wise, whoever is interested can participant, with that it also shows their organizational and managerial capabilities which help them easier to join the MT.

Also letting the QAH check objective and unrankable stuffs in qualified is a good point, and I think more weight of BN scores would be given to them based on the quantity they checked(encourage them). Meanwhile, if a map with unrankable issues went to ranked, the QAH who checked would get punished. The QAH is good to become a reward-more and punish-more role, they take the responsibility.

I believe with the new system change, the MT has many things to do at the beginning, like handle conflicts between different BNs and mappers after a DQ, moderation of the map discussion.
Last edited by Fycho , edited 4 times in total.
Spkz
Global Moderator
163 posts
Joined July 2013
Spkz
The only thing which came to my mind right after I finished reading the draft is the following question:

Draft wrote:

Applications for joining the Beatmap Nominators will mostly stay the same but instead of holding them every three months, they will be open at any point in time.

The applications will still be composed of two parts, a ranking criteria proficiency test and a manual evaluation by members of the “Management Team”.


So, if the tests will stay, how are they going to look like? Or rather, how frequently are you going update them? Because there were cases where people got accepted with others' help (revealing the whole test and the answers and similar) with the test. The only solutions which come to mind is whether you come up with a dynamic system or you update the tests pretty much daily / or you make seperate tests for each individual applicants (or even more than one, given the chance they might fail). The first option would be far more ideal obviously but I can hardly imagine a possible way to do it, the second option is well....time consuming and redundant so it is probably not the way it should go as far as I can tell.

Now, obviously you mention that the "implementation of this system might take some time to prepare" but it is probably worth the time to play with the thoughts before you actually start making this system becoming real.

Also, please keep in mind that I'm fairly outdated with how BNs get promoted nowadays (thus I don't really know how a "proficiency test" really looks like today if those have been changed) so please ignore me if my question didn't make any sense in the current context.
Last edited by Spkz , edited 2 times in total.
Yugu
Elite Mapper
1,006 posts
Joined August 2013
Yugu

UndeadCapulet wrote:

kowai
timemon
Beatmap Nominator
1,307 posts
Joined November 2012
timemon
My biggest concern is ability to DQ as a Beatmap Nominator.
To be entrusted with an ability to disquality beatmaps requires a special kind of trust. Giving such ability to a big group of people sounds like a recipe for disaster without proper safeguard.

Either give BNs ability to mark beatmap for Disqualification but have the actual action done by the Management Team to protect against foul play.

Or in order to DQ a beatmap, it requires multiple BNs to hit the DQ button enough times to prevent a bad apple going berserk on the qualified section.
camellirite
215 posts
Joined February 2015
camellirite
all BNs getting DQ ability is pretty risky. I think a different, more trusted group should handle DQs or requests for DQs. Maybe BNs above a score threshold, or the 10 best scoring BNs have the ability to DQ?

or maybe even have a DQ button for all BNs which has to have a reason attached to the request, and trusted members accept or decline the request?
Mordred
532 posts
Joined October 2015
Mordred
tbh I don't really see why we need to force a change when one is simply not needed, it seems like someone saw shiten and was like "oh qat sucks lets do something else". This new system is just gonna be the same thing except more annoying to deal with

Proposal wrote:

Disband the Quality Assurance Team
While I totally agree with changing the name (since lets be real, qat hasn't assured quality in forever), I don't think disbanding it entirely is necessary though.

Proposal wrote:

Give the Disqualify button to the Beatmap Nominators
A little concerned with this part, not about the potential of abuse or anything but more about the fact that "a map cannot be disqualified for anything but unrankable or objective issues". Might as well say only unrankables since that's the only truly objective issue, nearly everything else is mostly subjective. This also allowes the mapper to simply disregard any dq mods whatsoever without having to even try to explain why they don't want to change things, why explain when you can just say no.
Aside from that, how is a BN supposed to act on any reports if they aren't for strictly unrankable issues? The only way would be to just go ahead and veto the map if you think the concerns brought up by whoever are valid and just dq the map for that.

That being said I like the idea of BNs having a dq button for unrankables, having to wait for a qat to show up every time is a little stupid lol. But the issue with this is that the dq button would most likely replace the problem button. There's a huge difference between posting something as a suggestion and something as a problem, a lot of people tend to look at suggestions like "oh whatever it's just a suggestion" and only take problems seriously. If this gets implemented we would definitely require an actual dq button instead of just replacing one of the existing ones.

Proposal wrote:

Vetoes will be decided by the Beatmap Nominators
Uhhh can we not? From what I've heard from a few qats it's already annoying as fuck to deal with, but the thing is if you become qat you kinda sign up for this stuff, forcing this on BNs who really don't give a shit about vetoed map #27 is not the best idea, most of them will probably just pass the map without looking at it because they don't feel evaluating a veto on a map they couldn't care less about.
On top of that, having every single BN vote on every single veto is obviously not going to work and I believe I've seen Mao and others already suggest to just randomly pick a select number of BNs for this. While this is a lot better than having everyone vote it still kinda sucks, what if you only pick BNs that hate / love that particular map (guren lol)? In the end I believe the current system we have for this is the best, despite being kinda not so good.

Proposal wrote:

Content related moderation will be handled by the Global Moderation Team
I mean sure but like do we really need this? Not entirely sure if most of the gmt even cares about anything mapping related, I highly doubt most of them will want to moderate map threads, hell even the current qat barely does that (which is why this proposal is a thing btw xD). I'm fairly confident things wouldn't turn out nearly as bad every time if there was some actual moderation for controversial map threads (guren, shiten, etc.) instead of just saying "pls behave" after it's already way too late.

Proposal wrote:

A new score system for Beatmap Nominators will be introduced
I think nao already mentioned something about that, but what this is gonna do is promote nominating the most boring maps you could possibly find. Why nominate a map with 200 redlines / a set with 15 diffs with a lot of potential unrankables / similar if you can just nominate 5 diff anime TV sizes all day long? The risk of fucking up isn't worth it for most people already, and changing things will not help with it. On top of that what if people just nominate subjectively terrible maps all the time, but they don't have any unrankables, would that still make them a good BN? I believe being BN is about more than just checking a set for unrankables (even though that's basically what the job description is but you get the point).
This would also require a clarification as for what would actually give you minus points, what counts as a major issues (small metadata mistakes should never be relevant for example) and all that kind of stuff.

Proposal wrote:

Beatmap Nominator Applications will always be open
When will applicants be evaluated though? As soon as they send their application or will there be a set date each month? Will the management team really be able to handle this and not have delayed results? I definitely support this particular point though.





In the end I disagree with almost everything except changing the name of the qat and always open applications. I still don't believe we desperately need to force a change, the current system works as it is.
hi-mei
784 posts
Joined June 2014
hi-mei
The proposal is pretty good, most of the things brought up I really like.

Tho what about bn tests? If you get rejected, how much of time you gonna need to wait till next reapply? I still think we do need to change something in that regards too. Like, judging and so on.

So after you pass a test on RC, the next one is a test of "Whether a judge likes you or not"
Cmon lets get rid of that

And also, in regards of BN apps, can we kinda move from this Three Month stereotype? Here is whats happening with these mods: first 2 months people are just abusing kudosu with low quality mods, then in last month they do good stuff and then they apply.

Can we just move it to 6 weeks just like exactly 1 cycle of BN app?
Last edited by hi-mei , edited 2 times in total.
Kibbleru
osu! Alumni
5,846 posts
Joined August 2013
Kibbleru

Spkz wrote:

-
So, if the tests will stay, how are they going to look like? Or rather, how frequently are you going update them? Because there were cases where people got accepted with others' help (revealing the whole test and the answers and similar) with the test. The only solutions which come to mind is whether you come up with a dynamic system or you update the tests pretty much daily / or you make seperate tests for each individual applicants (or even more than one, given the chance they might fail). The first option would be far more ideal obviously but I can hardly imagine a possible way to do it, the second option is well....time consuming and redundant so it is probably not the way it should go as far as I can tell.

Now, obviously you mention that the "implementation of this system might take some time to prepare" but it is probably worth the time to play with the thoughts before you actually start making this system becoming real.

Also, please keep in mind that I'm fairly outdated with how BNs get promoted nowadays (thus I don't really know how a "proficiency test" really looks like today if those have been changed) so please ignore me if my question didn't make any sense in the current context.



The tests are actually pulled from a pile of a bunch of random questions. So i believe we will just delete / create new questions every so and then?
Monstrata
Elite Mapper: Aspirant
4,642 posts
Joined May 2013
Monstrata

Mordred wrote:

Proposal wrote:

Vetoes will be decided by the Beatmap Nominators
Uhhh can we not? From what I've heard from a few qats it's already annoying as fuck to deal with, but the thing is if you become qat you kinda sign up for this stuff, forcing this on BNs who really don't give a shit about vetoed map #27 is not the best idea, most of them will probably just pass the map without looking at it because they don't feel evaluating a veto on a map they couldn't care less about.
On top of that, having every single BN vote on every single veto is obviously not going to work and I believe I've seen Mao and others already suggest to just randomly pick a select number of BNs for this. While this is a lot better than having everyone vote it still kinda sucks, what if you only pick BNs that hate / love that particular map (guren lol)? In the end I believe the current system we have for this is the best, despite being kinda not so good.
I feel like if BN's don't want to be involved in vetoes, they shouldn't have to. But then they hold a "non-opinion". Let the people who care about vetoes do the vetoing, and let the other BN"s who also care about vetoes handle the veto mediation.

If you don't care, it is no loss to you to ignore it imo. If the system becomes such that a lot of maps you think are fine are now being vetoed, then maybe it's time for you to step in and start countering them during mediations.
Monstrata
Elite Mapper: Aspirant
4,642 posts
Joined May 2013
Monstrata
I don't really get why some of you guys are so concerned about BN's being able to disqualify. You've been able to qualify maps forever. How hard is it to just not hit disqualify unless you see objectively unrankable issues? Involving Management Team for every single DQ requires additional communication that imo is not necessary when you see an obviously offscreen slider, etc... (And these are really the only times you'd DQ without mapper's consent anyways).

The only time a BN can DQ without mapper's consent and without it being unrankable issues, is if the BN is placing a veto. And imo, I feel it should be mandatory for BN's who want to DQ a map for subjective issues to first give notice (24 hours) for mapper to respond. Therefore, I think it is appropriate to say maps can only be DQ'ed by a single BN if the BN plans to veto the map, and the mapper either hasn't responded in 24 hours, or has not given the BN an adequate response in 24 hours. For cases outside of this (past the 5th/6th day), I think the support of 2 BN's is necessary in order to DQ a map on like the 6th or 7th day for subjective issues. (This would basically be two BN's completely negating the nominations of the previous two).
Kibbleru
osu! Alumni
5,846 posts
Joined August 2013
Kibbleru

Monstrata wrote:

I don't really get why some of you guys are so concerned about BN's being able to disqualify. You've been able to qualify maps forever. How hard is it to just not hit disqualify unless you see objectively unrankable issues? Involving Management Team for every single DQ requires additional communication that imo is not necessary when you see an obviously offscreen slider, etc... (And these are really the only times you'd DQ without mapper's consent anyways).

The only time a BN can DQ without mapper's consent and without it being unrankable issues, is if the BN is placing a veto. And imo, I feel it should be mandatory for BN's who want to DQ a map for subjective issues to first give notice (24 hours) for mapper to respond. Therefore, I think it is appropriate to say maps can only be DQ'ed by a single BN if the BN plans to veto the map, and the mapper either hasn't responded in 24 hours, or has not given the BN an adequate response in 24 hours. For cases outside of this (past the 5th/6th day), I think the support of 2 BN's is necessary in order to DQ a map on like the 6th or 7th day for subjective issues. (This would basically be two BN's completely negating the nominations of the previous two).


Additionally if it's clear that you are abusing DQ's then obviously the management team would do something about it?
Monstrata
Elite Mapper: Aspirant
4,642 posts
Joined May 2013
Monstrata
@kibb Yep, exactly. I feel like it's not too hard to implement a 1-dq per day limit into BN functions too (similar to 3 qualifications per day etc...) so we don't have CDFA v3 lol.
Kibbleru
osu! Alumni
5,846 posts
Joined August 2013
Kibbleru
The one thing i want to avoid is a scoring system.

As someone who had first hand experience with the scoring system, having that number out there which determined your outcomes just put pressure on us.

While i do think a scoring system is useful for the evaluation of BNs, it's probably better to keep it private, so BNs wouldn't feel pressured by this number. The management team will simply use that scoring system for judgement of said BN.


Monstrata wrote:

@kibb Yep, exactly. I feel like it's not too hard to implement a 1-dq per day limit into BN functions too (similar to 3 qualifications per day etc...) so we don't have CDFA v3 lol.


I'm honestly not too concerned about cases like these, since they happen quite rarely, and are quite easily revertable, there isn't much damage one can do by chain dqing a bunch of maps anymore anyways since the timer doesn't reset.

And honestly, yolo qualifying a bunch of maps is something you can absolutely do right now, yet i don't see anyone doing it. So what's the problem?
Last edited by Kibbleru , edited 1 time in total.
Aurele
Global Moderator
7,514 posts
Joined January 2011
Aurele

Kibbleru wrote:

The one thing i want to avoid is a scoring system.

As someone who had first hand experience with the scoring system, having that number out there which determined your outcomes just put pressure on us.

While i do think a scoring system is useful for the evaluation of BNs, it's probably better to keep it private, so BNs wouldn't feel pressured by this number. The management team will simply use that scoring system for judgement of said BN.


I am totally seconding this, as a victim of the scoring system back then, it is very pressuring for nothing.
Ephemeral
Inland Empire
3,945 posts
Joined April 2009
Ephemeral
any sort of progression metric will be kept private and use for application analysis only, yes. people won't be outwardly judged by their numbers, that's just stupid.

also helps to address the elephant-in-the-room of people gaming the metric for easy BN ship if they have no idea how it works :D
Last edited by Ephemeral , edited 1 time in total.
squirrelpascals
984 posts
Joined March 2015
squirrelpascals

Kibbleru wrote:

While i do think a scoring system is useful for the evaluation of BNs, it's probably better to keep it private, so BNs wouldn't feel pressured by this number. The management team will simply use that scoring system for judgement of said BN.


This sounds like it might be unpopular, but just knowing a number exists in the first place will put pressure on me, and I feel like not knowing what it is would only make that feeling worse. I would know that there is some sort of value that determines how i rank up against other bns, but assuming that we're not able to know what value that is (because it's kept secret), I won't know what exactly I'm able to do to perform better, forcing me to be perfect in every aespect (which isn't realistic). This sounds very similar to my experience that I explained on probation here.

I think a good middle ground for this would be making the viewing of the score sheet optional, because I imagine everyone will feel differently about being able to see their score :p So if a bn wants to see their score, they can request an invite to a google sheet that shows it or whatever, but if not that's ok too.
Last edited by squirrelpascals , edited 1 time in total.
Kibbleru
osu! Alumni
5,846 posts
Joined August 2013
Kibbleru

squirrelpascals wrote:

Kibbleru wrote:

While i do think a scoring system is useful for the evaluation of BNs, it's probably better to keep it private, so BNs wouldn't feel pressured by this number. The management team will simply use that scoring system for judgement of said BN.


This sounds like it might be unpopular, but just knowing a number exists in the first place will put pressure on me, and I feel like not knowing what it is would only make that feeling worse. I would know that there is some sort of value that determines how i rank up against other bns, but assuming that we're not able to know what value that is (because it's kept secret), I won't know what exactly I'm able to do to perform better, forcing me to be perfect in every aespect (which isn't realistic). This sounds very similar to my experience that I explained on probation here.

I think a good middle ground for this would be making the viewing of the score sheet optional, because I imagine everyone will feel differently about being able to see their score :p So if a bn wants to see their score, they can request an invite to a google sheet that shows it or whatever, but if not that's ok too.



Well, believe it or not, a score exists already (sorta), it's just the number of nominations, dq's, etc. you get. almost everyone is judged on a case by case basis, we look carefully into the reasons and circumstances, that "score" is only used as a guideline.



squirrelpascals wrote:

Mao wrote:

We also still need to figure out how to handle probation exactly.


For starters, I'd like to suggest telling the bns on probation generally what Management will be looking for when they're evaluated. My readmission felt a bit like the QAT was saying "ok you're a bn now but you're also on probation, so yeah gl." There wasn't really much guidance as to how anyone on probation was being evaluated. Were the QAT looking for number of icons? bubble pops or dqs? activity? I got a lot of mixed responses and some rumors from people I asked (bns and like 2 qat i think). For myself I had the reasons from my previous probation failure to focus on improving but I didn't see any clarity as to how our performance was being measured for the other probation bns. This just put pressure on me to be perfect in every aspect of modding, which I know isn't the true expectation of the QAT.

This whole ordeal made probation super super stressful for me, and it made it a bit of a mystery as to why some of the people didn't pass. Probation doesn't need to be stressful or mysterious. We need to write some sort of probation quidelines or something.


Interesting point, I've actually never thought this was an issue since I assumed people knew that they should just not get any dq's and it'll be fine
Last edited by Kibbleru , edited 1 time in total.
Monstrata
Elite Mapper: Aspirant
4,642 posts
Joined May 2013
Monstrata

squirrelpascals wrote:

Kibbleru wrote:

While i do think a scoring system is useful for the evaluation of BNs, it's probably better to keep it private, so BNs wouldn't feel pressured by this number. The management team will simply use that scoring system for judgement of said BN.
This sounds like it might be unpopular, but just knowing a number exists in the first place will put pressure on me, and I feel like not knowing what it is would only make that feeling worse. I would know that there is some sort of value that determines how i rank up against other bns, but assuming that we're not able to know what value that is (because it's kept secret), I won't know what exactly I'm able to do to perform better, forcing me to be perfect in every aespect (which isn't realistic). This sounds very similar to my experience that I explained on probation here.

I think a good middle ground for this would be making the viewing of the score sheet optional, because I imagine everyone will feel differently about being able to see their score :p So if a bn wants to see their score, they can request an invite to a google sheet that shows it or whatever, but if not that's ok too.
Scoring has always existed no?

I feel like it would be helpful for QAT management team to let BN's know if they are in danger of not meeting a certain internal scoring threshold, or are in danger of being put in probation/kicked. From what I'm seeing with the rework, Management Team will have far less work to do now, since anything DQ and veto related is no longer their responsibility, so having the extra task of letting BN's know they need to step up their work would be a good new task to give them I think.
Lumenite-
630 posts
Joined April 2015
Lumenite-
i've skimmed the thread, but i may have missed it-allow me to voice some concerns in a really, really nerdy way (and let me make clear i speak for taiko and taiko only)

i believe this proposal in giving BNs the DQ button and forcing all nominators to vote on a veto will result in a majoritarian democracy and tyranny of the majority in which if you find yourself in the minority over any subjective issue, you're tough out of luck. perhaps it may be a bit pessimisstic and paranoid, however i think this will create a war of pulling in your friends that agree with you into the BNG so they can agree with you and you can win vetoes.

i firmly believe that subjective more complicated issues that may or may not warrant a DQ should be left in the hands of those who are experienced in their knowledge. i can get behind all BNs allowed to DQ unrankable issues to keep things moving, however subjective things should be left to those who are experienced.

i think a good talking point to prevent the tyranny of the majority is to require a supermajority-or 2/3 vote-in order for a veto to be sustained or overruled. along with that, assuming probation would still exist, probationary BNs should only be allowed to DQ maps with unrankable issues, not subjective ones. those who have shown questionable judgment or are new to the BNG don't have the experience necessary to mediate in complicated issues, in my opinion.

if anyone has expressed these ideas or if there is a need for clarification, pls be nice to me c:
Hydria
Beatmap Nominator
1,245 posts
Joined April 2011
Hydria

Kibbleru wrote:

Interesting point, I've actually never thought this was an issue since I assumed people knew that they should just not get any dq's and it'll be fine


Nominating two generic maps with 0 DQs in probation is a lot different than Nominating 6 maps, 5 of which are unique, with 1-2 DQs in probation. You punish the one that puts in the extra work.

This system will be similar to the old Beatmap Nominator Rankings back in 2014 and 2015. However, instead of focusing on subjective issues, this system will only account for unrankable and objective issues.


I get that you're trying to avoid penalising mapsets that are controversial, but at the end of the day this change will still hurt those BNs, as you're objectively more likely to miss an objective mistake in something that's unique and controversial (or in a new mapper's first ranked set) than you are in someone that's profound in mapping pushing another 1-2 jump map.
Regardless of if BN removals are being judged on a case-by-case basis, this change will still deter BNs from going for controversial mapsets. People don't like being watched and judged with a number.
It's better to track someone's progress based on overall performance and not just objective dqs, even subjective dqs can need punishment if they're unwarranted and highly disagreed upon. Have people there that is qualified to track community progress and shifts to make the call on dq influence since they will always be more than a number.
show more
Please sign in to reply.

New reply

0
1 / 105