Hello everyone! After some discussions about the QAT rework thread, we are happy to present you our proposal for a new system.

Proposal [Updated] | Proposal

Please note that this is still a proposal and that we are taking feedback. If you want changes to be made to it, please post in this thread until Monday, February 4th 23:59 UTC+0. After that, we will try to incorporate your feedback as well as we can!

kowai

uh

Sounds interesting, but i have a little problem with this part:

Applications for joining the Beatmap Nominators will mostly stay the same but instead of holding them every three months, they will be open at any point in time.
The applications will still be composed of two parts, a ranking criteria proficiency test and a manual evaluation by members of the “Management Team”.

If you are removed from the Beatmap Nominators, you will be put on a cooldown of three months before you can re-apply. If you leave the Beatmap Nominators on your own merit though, you will be allowed to re-join within the next two weeks after your removal.

Okay so, sounds not bad but there is one aspect not explained. What if someone applied and got rejected. Will this person be set on a cooldown before they can re-apply or not?
It's a pretty important part. If there is no cooldown, people who got rejected and "Do not accept the decision" or pretty stubborn persons would spam their applications all the time and the "Management Team" would have to re-evaluate this person all the time new.

this is v nice i am pogchampioning out of my seat

Give the Disqualify button to the Beatmap Nominators
The Beatmap Nominators will take on the current QAT’s job of acting on reports in the Report a Qualified beatmap here! thread.

Vetoes will be decided by the Beatmap Nominators
Instead of a small group deciding about the fate of a map, the “Management Team” will initiate a majority vote among all BNs (except for the one who placed the veto) of the respective mode.

Top 10 plans to fail within weeks, guaranteed barely anyone will be trying to dq controversial or even just subjectively bad maps by april because it won't be worth the time and effort to do so.

You should clarify "If you are removed from the Beatmap Nominators, you will be put on a cooldown of three months before you can re-apply." to include "or your application is rejected" since that's presumably how the system would work.

I am concerned about 2 things. first: majority vote for all vetoes. I think this would be a lot of overhead and think that randomly selected pool of x amount of BNs would be better to avoid having to ping every BN and get their opinion on vetoed maps over and over, which would get tiring. Think of it like jury duty or something. If people are always asked "is this map which you really don't give a shit about rankable" then it will lose a sense of meaning resulting in people not looking at the maps or vetoes in detail and just saying yeah it's fine whatever.

second, and more importantly: unrankables and objective issues are not actually important or a good representation of a BN's skill. for example: BN A consistently pushes maps which have complex timing and snappings, which occasionally get DQ'd for some adjustments to timing or snapping. BN B exclusively nominates single bpm NHI 1/2 based anime maps which never get DQ'd.
BN A is at much, much, MUCH higher risk of getting kicked than BN B despite nominating more diverse or interesting content than the usual fare, and neither of the two maps getting DQ'd for being bad maps. Only counting objective DQ's does not mean that people can nominate risky maps - it just changes the definition of risky from "uses weird techniques" to "has an unquantized mp3" and doesn't address the core issue of people only wanting to nominate single bpm 1/2 based anime maps to avoid getting kicked for timing DQ's.
This is a fundamental problem with how QATs gave up on caring about map quality and only focusing on unrankables - unrankable issues are not major problems for the most part. A wrongly snapped slider takes a grand total of 2 seconds to fix. A 10ms red line adjustment is not a major problem that causes a good map to become terrible. Meanwhile people nominate all forms of complete and utter trash (Uta intro ver anyone?) which don't have unrankables and therefore suddenly they are fine.
I'm not saying to magically start giving a shit about mapping quality but this idea that a BN getting DQ's for timing is worse than a BN not getting DQ's because they nominate rankable trash is really stupid and shouldn't be in the score system. The score system should just be activity and SERIOUS unrankable issues (entire sections missing hitsoundings, large amounts of unsnaps due to fucked up green lines, 20+ms offset issues, etc.)

All I needed to read.

The “Quality Assurance Helpers” sub-group inside the BNs will continue checking every qualified map for unrankables and objective issues until an automated method replaces them. Moreover, the QAH will be renamed to avoid confusion with the former QAT.

The Beatmap Nominator rules will be updated so that a map may only be disqualified if unrankable or objective issues are present, the mapper requests it, or in the event of a veto.

Does this mean that probation BNs will also be able to DQ maps?

Beatmap Nominator Applications will always be open
Note that implementation of this system might take some time to prepare

Applications for joining the Beatmap Nominators will mostly stay the same but instead of holding them every three months, they will be open at any point in time.

Having the applications always open means that there would always be probation BNs, so would probation just disappear? Because I don't know how it could be done to be constantly looking over new people that don't come in a cycle or any set number, just random.

Just sounds a little confusing right now. ):

You guys should add a cooldown between how long people can apply for BN - perhaps two weeks or something. People could just spam requests every other day under this proposal since it's not listed...

edit: Hi Feerum! I didn't reload my page until after I posted this and you mentioned this already.

Maybe make the name something more based towards mapping? "Management Team" seems to general and reminds me of some office job lmao.

Even "Mapping Management Team" would be better imo.

Main thing I took away is that it’s just the current system but everything’s appropriately labeled instead

T_T

Re:vetos and votes

Will every BN be required to vote?
Are BNs allowed to abstain from the vote?
If yes, and a large majority of BNs opt to abstain from the vote (so that a very small portion of BNs actually voted on the map), what will happen?
Will probation BNs vote on vetos and if so, will their votes be as evenly weighted as full BNs?

Main Issues:

Majority Vote regarding Vetoes

It isn't feasible to hold a vote across the entire BNG. Something like selecting x amount of readily available BN's would be better. I think 7 is the most feasible. Selection should be random, of course, and participation in vetoes should become a mandatory responsibility of BN's. Currently, it seems as if people who don't want to participate in votes can just not. There is no penalty for not participating, but as a result the vote may be skewed. Therefore, in order to ensure participation, I feel BN's who are chosen to participate in any veto mediation vote must also provide a short opinion on their choice (yes or no). I think it only has to be 2-3 sentences, nothing major, but this is to at least acknowledge that the BN has participated in the veto and isn't just going to blindly vote yes/no to everything and be non-toxic. (Voting no to a veto mediation is NOT being toxic, let's not set any bad precedents).

What happens after a Veto?

This one's rather straightforward. After a veto has been placed, and has been maintained through voting, the mapper is then forced to change the pattern or leave the map to be graved. Who will lift the veto? I'm assuming the original BN. But what if they continue to maintain that the mapper hasn't fully addressed the issue, or the mapper continues to call the veto'ing BN back to recheck without completely fixing the issue?

I want to add two additional rules to flesh out veto mediation scenarios:

- After a veto has been maintained, If the veto'ing BN does not recheck (and/or approve) the changes made to the map one month after the mapper has called them back for a recheck, the veto will be lifted. This will resolve both issues. It allows for mappers to "escape" a veto if the BN is not willing, or has grown tired of enforcing the veto and constantly rechecking the map. However, it also allows the BN time before having to recheck. If the mapper continues to call the BN back without addressing changes adequately, the BN is also welcome to take other requests, ask the mapper to rethink their concepts, and recheck a few weeks later. I think this is fair, as some vetoes for core issues are not fixable without major reworks anyways.

- If the veto'ing BN is removed, any of the BN's who voted "no" in the veto mediation can replace the BN and uphold the veto. If no one wishes to do so, then after 1 month, the veto will be lifted.

How to get into Management Team?

I'm sure a few people (hi Ascendance) will be interested primarily in this. Is there a process to get into Management Team? And what will happen to other aspects of the mapping/modding community? I'm talking about pushing forward new amendments and changes to the Ranking Criteria mainly. (Which is another subset issue). How will people be chosen for this specialized team?

BN Acceptance

Will there be any change to the way BN's are accepted going forward, now that reviews are no longer every 3 months but on a rolling basis? Will activity count be lowered etc...? I'm also interested in changes to selection criteria. Seeing as how this can potentially mean a lot more BN's entering more frequently, (as well as exiting).

Also if someone is rejected, what is the cooldown before they can reapply again?

Another note: I feel severity of a kick should be considered too. For example, BN's kicked for missing too many unrankable elements, or are inactive etc... should not be subject to a 3 month cooldown imo. I feel that for unrankable elements, this suggests the BN needs more training, or isn't thorough enough, but 1-2 months should be enough. For inactivity, well, can't a BN just resign if they know they will get kicked for inactivity? Then they would be able to reapply 2 weeks later. On the other end, I think behavior-related kicks could be 3 months. I think having a flat # of months is not necessary, and you guys could change the cooldown to suit the specific case of the BN being removed. Basically, make cooldowns more case-by-case and not necessarily do 3 months for everyone.

Another change ugh..
I'll just follow along, dunno what to say..

why is BN score based on issues rather than # of maps ranked?

i don't like the idea of giving the DQ button to all nominators but everything else seems okay i guess.

I like where this is going at the moment.

However, I am having some concerns.

New Beatmap Nominators // Probation
How are new Beatmap Nominators and Probations going to be handled? Will they be able to participate in disqualifications, vetoes and majority votes?

Application cooldown
- Is there any cooldown for potential Beatmap Nominators to re-apply after they fail the test? If so, how long would it be?
- Just a clarifications, is there a cooldown for nominators leaving by their own? The follow sentence is a little confusing to me.

"If you leave the Beatmap Nominators on your own merit though, you will be allowed to re-join within the next two weeks after your removal."

Does it means that a nominator leaving on their own will only have two weeks to come back to the team? If so, this short period of time seems unnecessary. In this case, it would only require the nominator to send in a "Leave of Absence" for said time. Instead, give the chance to the nominator to join back in before the next 30 days after their removal.
Or does it means they can join back 2 weeks after their removal?

dudehacker wrote:

why is BN score based on issues rather than # of maps ranked?

beatmap nominators are in charge of, in essence, deciding what maps get ranked. therefore, under this, a bn doesn't actually need to have a ranked map, they just need to know what is high enough quality for ranked, and be able to spot issues within a map

tl;dr - a bn doesn't need to be a good mapper, but they do need to be a good modder

Proposal wrote:

The Beatmap Nominator rules will be updated so that a map may only be disqualified if unrankable or objective issues are present, the mapper requests it, or in the event of a veto.

So does this mean that "dqs for discussion" are essentially dead? This means that if the community is still discussing a controversial map that gets ranked in one day, nothing is stopping it from getting ranked. Unless a bn initiates a "veto for discussion?" Which sounds like a complete work around and totally separate from a veto's true purpose. Controversial maps are controversial for a reason, so I think that room for discussion amongst the community is important so that we can reference to it for similar future maps.

Proposal wrote:

A new score system for Beatmap Nominators will be introduced
However, instead of focusing on subjective issues, this system will only account for unrankable and objective issues.

It doesn't sound like it makes any sense to score bns only on unrankable issues they find, since that doesn't really consider the initial quality of the maps modded, which can give a false perception of a bn's activity.

Say, for example, bn #1 checks 20 maps that are all from fairly experienced mappers who know what they're doing, so he finds about 5 unrankable issues in total. bn #2 likes to help newer mappers out who are becoming more capable of ranking stuff, so he checks 4 maps in total and finds 10 unrankable issues. Assuming all the unrankable issues of all maps are found, bn #1 is pushing 5x more maps than bn #2 is through the ranking process. But under this scoring system bn #2's score is still 2x higher.

I've tracked all the unrankable issues I've found under my modding spreadsheet which needs to be updated, and using data from the past 5 months I found that the average amount of unrankable issues I find in each map comes to ~2.5. So I think a formula we can start from, measuring the amount of unrankable issues found per map, can look something like

(#total unrankables / #maps modded) / 2.5
So a score of 1 means you found an average amount of unrankables per map. Maps with no unrankable issues (from more experienced mappers) and maps that have a lot of unrankable issues (from newer mappers) will pretty much cancel eachother out.


I wrote a lot but I get this is still being developed :p Besides for those 2 things this proposal sounds super dope!

@dudehacker they never want to rank bns based on how active they are because then all the inactive people start yelling about how the active ones just yolo rank everything instead of thinking about quality =D
also they will never reward bns, so having a system that is positive (# of maps iconed) makes no sense because the only thing bns will ever get is punished; a negative system (# of maps dq'd/popped) fits that much better

I think that even though BNs will have the power to veto maps, I am especially worried that mini-game BNs will not use this power as often as they should due to either disinterest or a fear of backlash. BNs will be in the same situation as QATs before them where they are afraid to do a veto or afraid to vote yes on veto votes for subjective issues. there is also a chance that circlejerk vetoes will happen.

The QAH role is not as clear as it should be either since their main job was to assist the QAT. If the QAH has to directly veto instead of report less people will want to join due to having much more pressure on them than the average BN. There isn't really any reward for BNs doing a good job with this new system, instead it punishes them with more opportunities for community criticism and higher chance of kicking imo.
so not sure that QAT must be removed. I'm still think the current system is pretty solid, since it is just a inside problem.

Monstrata wrote:

It isn't feasible to hold a vote across the entire BNG. Something like selecting x amount of readily available BN's would be better. I think 7 is the most feasible. Selection should be random, of course, and participation in vetoes should become a mandatory responsibility of BN's. Currently, it seems as if people who don't want to participate in votes can just not. There is no penalty for not participating, but as a result the vote may be skewed. Therefore, in order to ensure participation, I feel BN's who are chosen to participate in any veto mediation vote must also provide a short opinion on their choice (yes or no). I think it only has to be 2-3 sentences, nothing major, but this is to at least acknowledge that the BN has participated in the veto and isn't just going to blindly vote yes/no to everything and be non-toxic. (Voting no to a veto mediation is NOT being toxic, let's not set any bad precedents).

Randomly selecting BNs was my first thought too though but in that case, we'd need to anonymise it as a fewer number of people is probably more prone to bribery. Also in that case the mapper must have a chance to contest the decision made by the BNs because if you are unlucky and get 7 people selected that all are in favour of the veto despite maybe most of the other BNs being against the veto, the decision would not be representative of the whole group.

Monstrata wrote:

How to get into Management Team?

I'm sure a few people (hi Ascendance) will be interested primarily in this. Is there a process to get into Management Team? And what will happen to other aspects of the mapping/modding community? I'm talking about pushing forward new amendments and changes to the Ranking Criteria mainly. (Which is another subset issue). How will people be chosen for this specialized team?

This has yet to be decided but I think it will probably go into the same direction as QAT promotions in 2017 pre-upheaval where when somebody shows good management skills (e.g. managing different projects well) and good behaviour (as they will most likely have the same permissions as QAT have right now) they will be brought up in the "Management Team" and a majority vote/discussion will happen.

Monstrata wrote:

Also if someone is rejected, what is the cooldown before they can reapply again?

Yeah, this has been an oversight and totally needs to be added. I think 3 months would work well as that's the current length of a cycle but I personally wouldn't mind reducing it either.

Monstrata wrote:

Another note: I feel severity of a kick should be considered too. For example, BN's kicked for missing too many unrankable elements, or are inactive etc... should not be subject to a 3 month cooldown imo. I feel that for unrankable elements, this suggests the BN needs more training, or isn't thorough enough, but 1-2 months should be enough. For inactivity, well, can't a BN just resign if they know they will get kicked for inactivity? Then they would be able to reapply 2 weeks later. On the other end, I think behavior-related kicks could be 3 months. I think having a flat # of months is not necessary, and you guys could change the cooldown to suit the specific case of the BN being removed. Basically, make cooldowns more case-by-case and not necessarily do 3 months for everyone.

The numbers in the proposal are still quite arbitrary and will probably be adjusted when we polish it. I like this idea though!

pimp wrote:

i don't like the idea of giving the DQ button to all nominators

What's the problem with that? The BN rules will be adjusted to forbid any kind of abuse and I believe we can trust BNs with another button, especially since it can't really break anything.

______

We also still need to figure out how to handle probation exactly. I have seen very strong opinions on this from both sides - while some people think it's very important to see if new guys are fit for the role and to give people who messed up another chance, others think it should be replaced by a system like the old three strike system where whenever you have a major messup, you get stricken and once you accumulated three strikes you get removed.
I'm not entirely sure which one of these works best with the proposed systems so I'd like to get some more thoughts on that here too.

Moreover, I wouldn't get too hung up on the score system yet as its direction heavily depends on implementation. I feel like the best way to go about it would be a system where both the amount of dqs, their severity and the BNs activity are calculated into a score and then once a BN drops below a certain threshold, a bot pings the "Management Team" and they'll look closer into that BN's case.

We will also try to keep you updated with the score system when concrete planning has started, i.e. there will most likely be a separate proposal for that in the future.

FrenZ396 wrote:

You guys should add a cooldown between how long people can apply for BN - perhaps two weeks or something. People could just spam requests every other day under this proposal since it's not listed...

edit: Hi Feerum! I didn't reload my page until after I posted this and you mentioned this already.

I believe that was the plan tbh. Should be 3 month cd after getting rejected pretty much.

Anyways, liking this proposal better than the other one

Very much for this proposal. It's basically taking the current system and making it match the reality of things, which is pretty nice to do.

My only gripes are basically entirely covered by nao in this post, especially that first point. That second point would be nice definitely, but u get the same issues with that as you get with QAT enforcing quality standards onto maps, so I don't expect anything to be done about that

Second this.

Actually I would suggest that the Management Team should also take responsibilities or initiatives regarding other mapping/modding-wise events to revive them, like Spotlight, Contest Organization, RC Amendment, Education and Mentorship etc. I believe some of these are in a stasis and/or require more hands. It will make the main tasks of the team to be more higher-level, and genuinely "management and leadership-wise". These events would actually, in my eyes, make the team do greater for the entire community than dq'ing over mapsets xd.