https://docs.google.com/document/d/1EbhpZviwcvAYM926IsKZ-Q8DKRH4N3_WdpYcD5JdD-U/edit?usp=sharing <-- post to read

https://twitter.com/Cheri_Osu/status/1369671204580306949 <-- Twitter post

On request by Ephemeral
-
community/forums/topics/1268711?n=14 <-- referring an QA extension/Adding a branch
community/forums/topics/1268711?n=15 ^ Me at the same time basically stated a similar thing in detail in a document

CONS ^ was similar to QAT and tried it before didn't work/don't want to repeat anything and lack of interest still

community/forums/topics/1268711?n=59 my adjustments on -White's post
Basically about making a third group w/no BN benefits for non-bns to do QAH to add incentive instead of relying on BNs who don't want to do it.
community/forums/topics/1268711?n=46 ^ the original

CONS ^ No point in doing this after being BN, Having to trust non-bns is risky even with considering they can't dq, not as reliable as a BN doing it, rewards can be considered iffy,etc.

community/forums/topics/1268711?n=85 <--- abolishing the system

community/forums/topics/1268711?n=126 ^ my tl;dr version why just abolishing it isn't the best idea

community/forums/topics/1268711?n=107 ^ the full version + just be me
explaining the same thing in my document a bit better

community/forums/topics/1268711?n=123 <-- refers to having QA as a group making RC proposals and such
community/forums/topics/1268711?n=130 ^ basically rejecting it as RC is fine as is

community/forums/topics/1268711?n=139 QoL changes could add

Since it's here too this is my response to the suggestion of splitting the BNG and QAT https://gist.github.com/VINXIS/25aeae000261f07ae1bc52f7f2c25043

tl;dr history suggests its a bad idea

A suggestion is a suggestion at the end + what happen in the past doesn't necessarily mean it shouldn't happen again especially since the game is much bigger now than it was back then and there was obvious a lot more mismanagement back that can be easily rectified if we look more into the core problems.

The suggestion at the end was nothing more my (and friend) 2cents on options to do, I imagine there is much more

I agree, we should be looking more into the core problems, I just felt it was necessary to respond to that and that the same thing will happen again because the environmental factors caused are still about the same because of those core problems

everything else in the post I think was good and how there has been so little action by the NAT either especially considering the amount of data that is pretty easily achievable that could be used to solve issues

VINXIS wrote:

everything else in the post I think was good and how there has been so little action by the NAT either especially considering the amount of data that is pretty easily achievable that could be used to solve issues

what sort of data do you mean here, exactly? is this in regards to punishing dq's or something else?

Hi, author of the first half of the google doc here

VINXIS wrote:

tl;dr history suggests splitting the BNG and QAT is a bad idea

I mostly came up with this because despite QAH being a volunteer position within the BNG, there is no real obligation attached to it, so there is no artificial motivation to perform QA as a BN (on top of the no/negative motivation to do so naturally). Splitting off the QAH/QAT/whatever you want to call it was what I envisioned as a rather extreme method of defining the roles of those who are actually willing to do this work (sure, there are probably none that are willing to do it to the extent that Cheri does, but I'm fairly sure there are plenty that would do QA at least to some extent if they knew it was to fix a system on the verge of collapse as things currently stand).

Theoretically, this would also be solvable with QA quotas for ALL BNs, but that would likely deincentivize BN applications at all, so not exactly an ideal solution. A large BNG is good with how big the game has gotten, but not good if the other critical systems in place (such as QA) can't even begin to keep up.

I recognize that the question of how to motivate QA activity is extremely difficult, but especially given the current additional load on the qualified section thanks to the mug event I think it's more important than ever that there is at least some serious discussion on what to do about it.

Some points of data I think would be beneficial off the top of my head

1. Nature of vetoes presented
2. Nature of the disqualifications in general
3. The states of qualified and ranked throughout this period of using such a system
4. Survey analysis

This is in regards to helping find out what may have indirectly/inadvertedly caused what when using the system

aidanbh wrote:

Splitting off the QAH/QAT/whatever you want to call it was what I envisioned as a rather extreme method of defining the roles of those who are actually willing to do this work (sure, there are probably none that are willing to do it to the extent that Cheri does, but I'm fairly sure there are plenty that would do QA at least to some extent if they knew it was to fix a system on the verge of collapse as things currently stand).

That's fair enough, though I do think the only time this type of change will be fruitful is when the treatment of the dq + discussion system is changed

VINXIS wrote:

That's fair enough, though I do think the only time this type of change will be fruitful is when the treatment of the dq + discussion system is changed

agree on this point, will discuss further at some point later today(?)

VINXIS wrote:

Since it's here too this is my response to the suggestion of splitting the BNG and QAT https://gist.github.com/VINXIS/25aeae000261f07ae1bc52f7f2c25043

tl;dr history suggests its a bad idea

You have a pretty fair point but it's important to note that the reasons the QAT was hated back then are rather easy to mitigate; qualified maps no longer give pp (which fueled a lot of player hatred towards disqualification) and often times QAT posts were very subjective and contentious (see: Asymmetry, first DQ is post 107 [did you know you can click the number and go to a specific post? I had no clue until now lol]). The pp debacle was settled a while ago, but I'd also argue that 'subjective DQ' debacle is effectively settled with the introduction of the veto (and mediation) system. As flawed as it is, it is a good replacement for the old "QAT says No thanks" system of before; now, a large portion of the BNG is required to give fairly detailed reasoning as to why a veto should or should not stand. A separate user group could just essentially be the 'janitors', clearing out obvious spread issues, straight-up missed unrankables etc.

I don't necessarily think that a new user group is the best decision, but it's definitely not as apples-to-apples as it might seem on the surface.

For the record, I entirely agree with the idea that the way we view disqualifications is the root cause, and I'd suspect that a lot of that has to do with the time aspect. You can wait days or weeks for a BN to find time for your map which you likely spent dozens of hours making, or you can search for days for BNs to wait for weeks, and by the time that the map is qualified, the absolute last thing you want to do is have a last-minute disqualification throw another wrench into the system. From a mapper's perspective (especially one without close ties to BNs), it isn't as much of mutually-beneficial quality assurance as it is just pure pain. Though honestly, I don't see that there is a fix for this issue without an extreme reform (like, no-BNG level of extreme) that, frankly, I'm not sure how it could possibly be done.

In my perspective at least as a mapper I think that if im given suggestions + stuff and im in a process where the discussions are not meant to be a benefit to the general ranking system, then the process of applying mods in qualified should be so that all i have to do is apply and update, no need for a DQ to apply and update, and Then have to wait for the bns to be free to check and nominate again after many days, and Then on top of that also have a punishment given to the bns just because I myself was receptive to feedback while in the qualified section that I thought would improve the map And were also not even unrankable issues Yo hailie can we get utsuroi qualified again yet, I think that that would at least to some nontrivial degree make it far less painful to be receptive to feedback in qualified for everyone, and also Wayyy back when the 'Fix Vetoes' Thread existed I was suggesting version history to make this far easier to do.

Also unrelated since this is 2 discussions at once, but I definitely think the 'subjective DQ' debacle is nowhere near effectively settled using the veto + mediation system because it is effectively the same process with the facade of a form of discussion occurring in the veto state and in the mediation state, which is why I was saying that the suggestion in the current environment of things would definitely not be good at all. I do agree that it is better than the shit QAT used to do for sure and removing pp and queue time reset has significantly helped, but I don't think that is enough for us to consider it environmentally different enough still because the root issues from then are still in effect today

nobody wants to do QAH work quite simply because its fucking boring and there's no incentive. you're not even doing an actual bn check you're just checking for unrankables and giving it a general pass of "yeah i guess its fine mapping wise"

speaking as someone who did a boatload of them in the past and will never do one again, I wish I could say something beyond "theyre boring" but it's just the facts. the fact that you still do them is honestly incredible to me (in a positive sense)

given that I don't see why anyone would actually want to be a QAT in your proposed system. its just being a BN but without any fun. and given that the compensation for the work will inevitably be nothing (as usual) nobody would want to do it lol

---

Also, given that mapping is a community sphere, I dislike how QAH work makes you look like an asshole if you dq any maps. people like to say "well the maps shoulda been good" and yeah you're literally right but we also live in a world with human beings that have emotions. nobody likes getting their map dq'd. being a QAHer puts you in a position where you're directly contributing to punishing your fellow BNs.

hard agree with how stupid and inconsistent the "severity" rating thing is

In response to hobbes:

giving some sort of incentive to do QA is an issue, and I do agree a lot of people would consider it boring, but I feel like that neglects the fact that some people enjoy things that are considered mundane or tedious to most. Since the community is growing so rapidly there are more people who would be willing to do said tedious tasks.

In regards to the comunity backlash aspect I feel reletivly the same way. Bigger comunity means more people who are able to not care about backlash.

----------------------------------------------------------------------------------------------

In general I feel like being less fixated on the singular proposed would be really beneficial for this entire ordeal since, at the end of the day it's just that. a suggestion from a singular person. It's sort of counterproductive to have such a strong unmovable stance in this sort of thread in my opinion since qualified is being held together by a single thread that's on the verge of snapping. New ideas, or ways to improve existing one's are the most important thing here.

----------------------------------------------------------------------------------------------

In regards to the proposed solution in the doc:

While I don't know if bringing back what is basically old QAT will be sustainable in the long term I feel like it's still worth discussing, even as a temporary fix while a more long term solution is worked on, especially since it doesn't seem like it would be overly hard dev side (may be wrong on this, so correct me if I'm wrong).

In my opinion old QAT would have worked if, one there was enough people who were willing to do the boring work and didn't care about community backlash, and two if there was some sort of way to moderate stupid decisions made by the QAT members.
I feel like the proposed solution of 3 teams (BN, NAT, QAT) could very well solve that issue. Some sort of objection function on the mapper's side that leads to discussion between NAT and the non offending members of the QAT, which could result in the offending party getting kicked if their DQ was too out of line. Having that out side pair of eye's (being the NAT) could potentially aliviate some to a lot of the bias in whether the DQ in question was out of line, and the appropriate punishment for the offending QAT member.

I also feel like the main reason why a lot the reason people who would otherwise be willing to check qualified don't is because being on a level playing-field with the people you disagree with is hard, so giving those people who would be willing to a role with similar power, or perceived power as NAT would make them more likely to check qualified.

Obviously this is just my two cents on the issue, but I strongly feel like something needs to be put in place to deal with qualified sooner than later, so going in circles on whether one idea will work really won't help.

well the alternatives are:
- slowing the entire ranking process WAY down in order to thoroughly check every map in qualified before letting it through. this would ensure quality, but would make a lot of people very angry as way fewer maps would be getting ranked and it would be tiring for those doing the checking. obviously not sustainable with the current scale of the game.
- ditch the QAH system entirely and just let stuff go through qualified (this is functionally what we have right now because as you said no one wants to do QA voluntarily). also not sustainable since as the game grows, so will the rate of major issues getting through qualified undetected.

so one of two things needs to happen:
- we find a group of people who, against all the classical theories of motivation, are willing and able to dedicate themselves almost exclusively to QAH work, and set them aside to do that specifically. I think Cheri proves that they exist, even if they are few and far between. it's also possible that a group of people exists who would not do it in a vacuum, but would do it with even a tiny bit of external motivation (such as the positional things mentioned in the document/by P1T)
- thoroughly rework some other part of the system to reduce the number of unrankables getting qualified in the first place. I know I said in the document that it was impossible to entirely eliminate human error in the BNG, but it may be possible, by raising the standards of BN performance, to get it down to an acceptable level such that QAH becomes unnecessary

honestly my take on this is -

issue is - right now qah isnt incentivised at all

also within the NAT we have quite a few people specialising in different things (think pishi & naxess compared to what noffy & seto do)

so what could happen to solve all this is have sub divisions to the NAT group (not seperate titles just offical different obligations within NAT) to include QAH work, so you have 3-4 NATs that their primary job is to do qah, 3-4 NATs that their primary job is to do new and current BN evals etc. (not delegating this among current NATs but promoting a few qah centric modders to fill these roles)

this would solve the issue of giving an incentive to do qah (getting nat leadership position) while also getting a more active and consistent userbase doing qah

i mean rn the qat probably dont want this change to be made - but its either this, make it mandatory for bns (which everyone will hate) or just not do qah at all (which seems counter-productive with the influx of new bns over the last two years or so)

i fully expect no changes to be made but with the current state the entirety of qah basically rides off of cheri doing the work - so what happens when she stops doing it?


also this would avoid trying to split up the bng which was tried before (think tier 1 & tier 2 bns etc)

https://docs.google.com/document/d/1BfOpVJt05aNobf0zoDTOFzgVyFbC7kzAPB4aj--QvGU/edit?usp=sharing Deviouspanda basically stated similar to what I was going for with this document but nonetheless i'm adding it here anyways since I was told to

All and all I simply just want to be able to not be the only one doing qualified and having solutions like above ^ would be a good thing even if temporarily to solved that problem until more reworks that will take much longer than the simple fix here.

Im not sure how it is for anyone else here but I think if I was in the position to, I would personally be more than down to do QA work or post mods on almost all maps in the qualified section myself if

1. it only focused on the aspect of improving the map and discussing with the mapper
2. it didn't have to affect anyone outside of the realm of the mapper and myself

which is why I wanted to suggest removing the dq punishment system and getting version history and everything instead of trying to "force" people to get into QA work, but idk how many more people feel similarly on working on QA so maybe making more people to QA work in the NAT or similar really would be the best option, idk tho

VINXIS wrote:

Im not sure how it is for anyone else here but I think if I was in the position to, I would personally be more than down to do QA work or post mods on almost all maps in the qualified section myself if

1. it only focused on the aspect of improving the map and discussing with the mapper
2. it didn't have to affect anyone outside of the realm of the mapper and myself

which is why I wanted to suggest removing the dq punishment system and getting version history and everything instead of trying to "force" people to get into QA work, but idk how many more people feel similarly on working on QA so maybe this really would be the best option, idk tho

I'm not denying that having such could be potentially helpful in the dear future as honestly I wouldn't even have to go through qualify the way I do if things was more laxed as such, but that is a system that takes work and still as well as still gotta considered that qa itself is a rather boring task on it's own.

My solution is basically to create management for the very core needed part of the game with the influx of mappers and nominators. So whilst yes a perfect world, going with solutions like this would be cool, but right now I think having something more QOL up until things are more stabled, should be prioritized.

fair enough, I just don't have much trust nor expectations for quick changes anymore because of how the past 14 years of osu have ONLY been just quick changes (and not just in the ranking system, in EVERYTHING related to osu) but it could be worth for now i guess


Edit: though at the very least the suggestion for raising QA activity value shuold really be done as well honestly

VINXIS wrote:

Im not sure how it is for anyone else here but I think if I was in the position to, I would personally be more than down to do QA work or post mods on almost all maps in the qualified section myself if

1. it only focused on the aspect of improving the map and discussing with the mapper
2. it didn't have to affect anyone outside of the realm of the mapper and myself

which is why I wanted to suggest removing the dq punishment system and getting version history and everything instead of trying to "force" people to get into QA work, but idk how many more people feel similarly on working on QA so maybe making more people to QA work in the NAT or similar really would be the best option, idk tho

Honestly I don't think getting rid of DQ punishment would be a bad idea, hell even just making it more lenient and consistent would help.

I feel like if there was quality control of some sort in place there would also be less need (excluding more extreme examples such as BNs who have a lot of generally agreed on questionable noms, but thats sort of case by case).

Some sort of QA being in place would likely encourage BNs to be a bit more careful in general so I feel like it very well might work.

Also having more people checking qualified would definitely make focusing more on improving the map and having discussion when it comes to subjective issues a lot more viable.

I feel like a lot of the issues (and you can correct me if im wrong here) people have with the way cheri DQs stem from the lack of time and support given to her to check qualified. Having more people doing the job would give more time to discuss, and provide more meaningful feedback/mods.

Checking qualified alone basically means to be able to get through everything and do more than check for unrankables DQing more immediately if the mapper doesn't want to change anything is more efficient, but given more time I'd assume she would prefer not to do that.

Yea i was talking about removing the DQ punishment/sev rating in the posts above since the focus for DQ punishments should really only be for bad actors, and i dont think someone modding a bunch of qualified maps improving the beatmaps and getting them changed shouldnt be considered as the BNs that nominated those beatmaps being bad actors for example, would even help with the current mods that cheri is doing in QA currently

VINXIS wrote:

Im not sure how it is for anyone else here but I think if I was in the position to, I would personally be more than down to do QA work or post mods on almost all maps in the qualified section myself if

1. it only focused on the aspect of improving the map and discussing with the mapper
2. it didn't have to affect anyone outside of the realm of the mapper and myself

which is why I wanted to suggest removing the dq punishment system and getting version history and everything instead of trying to "force" people to get into QA work, but idk how many more people feel similarly on working on QA so maybe making more people to QA work in the NAT or similar really would be the best option, idk tho

Figured I'll mention my experience as just a modder and never official staff: I personally enjoy looking at maps in Qualified but I'm almost always super reluctant to mention anything because the last thing I want to do is stir up shit where it isn't deserved (tho sometimes it is deserved NGL) and end up wasting the mapper's time, and as the system currently is the mapper is kinda incentivized to be as resistant to mods in Qualified as possible in order to not lose days of time to arguments, so it's kinda a lose-lose situation either way.

I'd be a lot more incentivized to mod (and look at) maps in Qualified if pointing out issues didn't have the chance of resulting in days-long wait periods, cuz that's just not fun for anyone at all

some thoughts on ongoing discussion

- remove DQ punishment and SEV rating (@VINXIS)
Very confused here, issues stated come across as exaggerated to me. Since we look at resets as a whole, BNs rarely (if ever) get punished for a single mistake but rather for having repeat issues or failing across the board. The other way around, a single DQ/reset in itself does not cause a BN to be punished. I'd hope at least BNs themselves understand this.
Similarly, as long as it doesn't fall under knowledge we judge on applications, feedback and improving maps is not harshly treated.

I don't know what "bad actors" are here, but regardless of being one BNs still have the objective task of making sure their nominations get ranked without issues. Removing "DQ punishment" and SEV rating essentially means to also remove accountability for this task being done well. Personally I struggle seeing how that works out okay in the long run, so I don't support it.

Improving the consistency of SEV rating we'll look into, nobody should be getting confused or afraid of these.

- QAT as a solution
Not sure but not a fan. Agree that repeating a failed history is something I don't want to pursue. I also don't see real suggestions that would improve on the old formula, lots of "maybe??"s here. The thought of surrounding context having changed enough to fix some of QAT's bad sides is appealing though.
Temporary QAT sounds like it'd become a permanent band-aid, knowing OSU. It'd need to be a transitional system but the other side of that hasn't been defined.

Otherwise I'm wondering what separates this potential QAT group from BNs in terms of ability and in terms of needed authority. What more is needed to check qualified maps and to offset the negatives that come with it.

- QAH
Boring, low interest, prone to backlash. Means it inherently needs an external motivation rather than relying on intrinsic efforts or inexplicable acts of altruism. Somehow something of value needs to be presented in return. Personally this is where I hit a dead end, we don't really have anything of value to offer (putting aside the QAT idea).

Maybe an improved star priority system could create such value? That's my best stab at introducing an idea of my own here.

Thinking about this topic has always put me between a rock and a hard place; it seems like there's never a good solution. So, I'm eager to see other people take a crack at it at the very least.

yaspo wrote:

some thoughts on ongoing discussion

- remove DQ punishment and SEV rating (@VINXIS)
Very confused here, issues stated come across as exaggerated to me. Since we look at resets as a whole, BNs rarely (if ever) get punished for a single mistake but rather for having repeat issues or failing across the board. The other way around, a single DQ/reset in itself does not cause a BN to be punished. I'd hope at least BNs themselves understand this.
Similarly, as long as it doesn't fall under knowledge we judge on applications, feedback and improving maps is not harshly treated.

I don't know what "bad actors" are here, but regardless of being one BNs still have the objective task of making sure their nominations get ranked without issues. Removing "DQ punishment" and SEV rating essentially means to also remove accountability for this task being done well. Personally I struggle seeing how that works out okay in the long run, so I don't support it.

Improving the consistency of SEV rating we'll look into, nobody should be getting confused or afraid of these.

- QAT as a solution
Not sure but not a fan. Agree that repeating a failed history is something I don't want to pursue. I also don't see real suggestions that would improve on the old formula, lots of "maybe??"s here. The thought of surrounding context having changed enough to fix some of QAT's bad sides is appealing though.
Temporary QAT sounds like it'd become a permanent band-aid, knowing OSU. It'd need to be a transitional system but the other side of that hasn't been defined.

Otherwise I'm wondering what separates this potential QAT group from BNs in terms of ability and in terms of needed authority. What more is needed to check qualified maps and to offset the negatives that come with it.

- QAH
Boring, low interest, prone to backlash. Means it inherently needs an external motivation rather than relying on intrinsic efforts or inexplicable acts of altruism. Somehow something of value needs to be presented in return. Personally this is where I hit a dead end, we don't really have anything of value to offer (putting aside the QAT idea).

Maybe an improved star priority system could create such value? That's my best stab at introducing an idea of my own here.

Thinking about this topic has always put me between a rock and a hard place; it seems like there's never a good solution. So, I'm eager to see other people take a crack at it at the very least.

Considering the whole entire ordeal of the problem is the fact we got nothing to work with and the idea of **QAT/Extension of NAT** would crack down on that more than what we have, a volunteer service with only one person doing. I truly disagree on the notion of denying this would be the best ideal situation just because of a claim "it happen before so we can't do it again" even after we explain in this thread why it is important being none of the current NAT care enough to do anything.


Completely dismissing the idea is such a bad move here when there is absolutely no incentive that the NAT has ever made since the time QAH was settle on the BN website and being that qualified is a mess, please understand this is close minded and rather only influence this issue to continue to happen.

As someone who checks qualified more than anyone else here, there is way more needed for a person who checks everything just as much as a regular NAT and I do not understand how does one not see that.

Moderation ability is always useful, supporter so we can download maps without having to pay for it, having a more of a shield from backlash (a title means more to people than you think), having a say in evals, etc.

Until you guys actually have a better solution to stop the mass backlog of 100 maps per week and relying on me to do so, I would like my idea of having this in the NAT to be in place (or something of similar extend)
or why am I even working this hard for people that don't seem to care here as you so willing to dismiss the idea so easily without any valuable idea yourself (that would work better here)?

yaspo wrote:

some thoughts on ongoing discussion

- remove DQ punishment and SEV rating (@VINXIS)
Very confused here, issues stated come across as exaggerated to me. Since we look at resets as a whole, BNs rarely (if ever) get punished for a single mistake but rather for having repeat issues or failing across the board. The other way around, a single DQ/reset in itself does not cause a BN to be punished. I'd hope at least BNs themselves understand this.
Similarly, as long as it doesn't fall under knowledge we judge on applications, feedback and improving maps is not harshly treated.

I don't know what "bad actors" are here, but regardless of being one BNs still have the objective task of making sure their nominations get ranked without issues. Removing "DQ punishment" and SEV rating essentially means to also remove accountability for this task being done well. Personally I struggle seeing how that works out okay in the long run, so I don't support it.

Improving the consistency of SEV rating we'll look into, nobody should be getting confused or afraid of these.

- QAT as a solution
Not sure but not a fan. Agree that repeating a failed history is something I don't want to pursue. I also don't see real suggestions that would improve on the old formula, lots of "maybe??"s here. The thought of surrounding context having changed enough to fix some of QAT's bad sides is appealing though.
Temporary QAT sounds like it'd become a permanent band-aid, knowing OSU. It'd need to be a transitional system but the other side of that hasn't been defined.

Otherwise I'm wondering what separates this potential QAT group from BNs in terms of ability and in terms of needed authority. What more is needed to check qualified maps and to offset the negatives that come with it.

- QAH
Boring, low interest, prone to backlash. Means it inherently needs an external motivation rather than relying on intrinsic efforts or inexplicable acts of altruism. Somehow something of value needs to be presented in return. Personally this is where I hit a dead end, we don't really have anything of value to offer (putting aside the QAT idea).

Maybe an improved star priority system could create such value? That's my best stab at introducing an idea of my own here.

Thinking about this topic has always put me between a rock and a hard place; it seems like there's never a good solution. So, I'm eager to see other people take a crack at it at the very least.

There really isn't any way to give external motivation for QAH.

There's the argument of "what if we just raise activity requirements, and make QAH checks worth a bit more", which sort of falls through because that will just lead to people doing more noms instead regardless of how much activity requirements are tweaked. The only real way you could make that work would be making QAH checks worth more than noms, and that basically leads back to essentially forcing people to do QAH, which people will get mad about.

Other than that the only other thing I could think of would be giving people who pass a certain check threshold a title as long as they keep up QA activity, but that likely won't work assuming the title doesn't come with any real perks, and if you're willing to give that title enough perks that it becomes worth going after you basically have a 3rd group at that point, which you disagree with.

There's no real clean solution to satisfy everyone, and I genuinly think having a 3rd group or a portion of NAT with the job of doing QA is the best option at this point.

yea

Regarding the "bad actors" statement I don't know what is defined as a bad actor outside of someone who like u said makes consistent mistakes, but I'd say that someone who is making consistent mistakes isn't the same as someone who doesn't complete the objective task of making sure their nominations get ranked without issues, but not sure about someone failing across the board

Exaggerated scenarios but

1. Someone who for example gets like 15+% of their nominations disqualified for only x issue will get punished since they are having repeat issues
2. Someone who had all of their nominations disqualified (let's say like 12, 4 each month in the 3 month period) for 12 completely different issues which were stated as suggestions given by people doing QAH work also will get punished because they are "failing across the board"

Is it intended to punish both or not? The response gives me mixed messages regarding that; if so, then what would be the reason to punish the second scenario if DQs similar to stuff like "changing soft sampleset to drum sampleset and raising vol by 10% for a section (lol deetz veto is 1/1)" or like "changing 2 sliders that unintentionally seemed out of design choice and caused a difficulty spike (this was 1/1 until I checked just now apparently it was rewritten and it's 0/0 now)" as examples occurred for all 12 nominations.

Is that really considered as failing across the board / not doing their objective task of making sure their nominations get ranked without issues? If not then nvm what I said this whole time loooool but I'd like to know what that actually does mean.

If so then this is my main issue because I don't think such a person should be considered as one that is failing their job as these are semi-unique cases and having that happen for all of them and be punished for that doesn't seem right to me at all. Everyone has their own accumulation of knowledge regarding mapping and some people are more knowledgeable about different stuff than others; sometimes the knowledge is conflicting and requires more discussion. I don't think that when stuff is treading these areas, there shouldn't be any punishment given for anyone participating

---


Oh edit, regarding the incentive of doing QAH work the easiest thing that can be done which was said above alrdy was just raising the value of QA work from 1/4 of a nomination tho i doubt thats gonna do much, could do something along the lines of some form of point -> reward system like a badge for example similar to mappers guild but then again its a bit aids to do stuff like this rn when responding to QA work on ur own map requires a lot of unnecessary work to

VINXIS wrote:

regarding the incentive of doing QAH work the easiest thing that can be done which was said above alrdy was just raising the value of QA work from 1/4 of a nomination tho i doubt thats gonna do much, could do something along the lines of some form of point -> reward system like a badge

admittedly I don't have much of a sample size here but the only active QAH said that badges and titles weren't really a motivating factor (and they have even turned down such bells and whistles in the past), and that the reason why a structured role/dedicated NAT position is not that is because such a position would come with a greater ability to actually exercise the job they are qualified to do - when looking at the kind of authority NATs command in beatmap threads as compared to regular BNs, one can see why such staying power would do well in the hands of QAH-type people.

I will admit that cheri is kind of a special case in this regard because the backbone she's built up from being one of the most hated figures in the entire game has made her fairly immune to people getting angry at DQs in thread (which has been mentioned multiple times as a major turn-off for QAHs) but I'm pretty sure there are a couple more people that would be willing to be brought on for this kind of role if it was in the form I just described

of course I could be entirely wrong about all of this but this is how I see the state of things from what I've observed/been told by multiple parties

Honestly, if the uncertainty of whether there are enough people in the current BNG who would even be interested, running a poll of some sort doesn't seem too out of the question.

Just make sure to explain some sort of (obviously vague) general expectation
(certain amount of checks need to be done per week, need to not care much about backlash, other stuff I cant think of right now)

If more people would like to do the qah job they would do it, just like some people mod outside the bng, the statistics show that some do but they're not even close to checking all daily qualified maps and that's normal since there are just too many, don't really think it can be solved tbh since the way people will see you won't change, you'll always be the bad guy that constantly checks maps looking for mistakes, also it's ultra boring and you gain absolutely nothing, volunteer jobs have their limits, i wouldn't expect people to perform consistently for free on this and tbh i don't see a way to reward them properly, most of what i've read here wouldn't change anything at all to make more people participate in qah and even less to make that participation consistent.

i'm chewing over a longer response to this once i'm confident i actually understand the issues at play here (presently 6 different people are talking about 6 different things). here's what i can suss out so far:

1. purportedly more bad maps are entering ranked because qualified isn't being checked
2. qualified isn't being checked because there's no real reason for anyone to check it
3. the only real reason to check qualified (QAH) is considered drudgework by 99% of the people who are in the position to do it, and thus don't do it (aka: QAH really fucking sucks and is a failed initiative)
4. mappers are reluctant to embrace feedback given during qualified phase because doing so imparts mandatory 7 day+ delays due to how the queue system works and other associated backlog with it, ergo they are punished for attempting to improve their map
5. the system as a whole (according to vinxis) pushes people away from actually making meaningful feedback because they get punished by current NAT practices for disqualifying maps with those issues
6. punishing people for leaving issues in maps provides incentive for people to hide/not talk about those issues, something that is becoming increasingly easier as the playerbase grows
7. all of this (and more) combines to paint disqualifications in an inherently negative light where nobody, not mappers, modders or players want to participate in the process
8. there is too much punishment and not enough incentive all around

is that about right?

@Eph
- i don't think "bad" maps getting ranked is due to qah inactivity as qah from conception is designed as a way to avoid unranks for cross screen 1/4 jump type things.***
- apathy towards checking qualified for unrankables can't be addressed because frankly bns have 0 incentive, reason to look at, or interest in maps other than ones they personally took a look at and wanted to push. fundamentally spending time sitting there looking at maps that don't interest you "for the greater good" of finding unrankables is a stupid process and one that peppy should have solved mechanically by now instead of dicking around with mania OD and stuff.
- mappers are reluctant to apply mods in qualified or even bubbled state because interacting with modders is a chore and relies too much on waiting for some arbitrary third party to sign off on work they aren't invested in compared to you and stalling a process which you put much more effort into than they did.
- the fact that 2 nats deciding to print a mug for $5 provoked the most activity out of bns in forever should tell you exactly what you need to know about carrots and sticks.

*** when people say bad, they are very rarely referring to maps with unrankables; people don't actually care about unrankables much. even things like spread quality aren't a huge factor in how people define quality offhand.
when people say bad, they mean either the top diff (or at least one of the top diffs) is subjectively low quality or the set as a whole is bad for meta reasons - pp farm diffs, low quality / low effort song, etc.
qah is not currently and was not ever designed to solve these issues, the whole bn system as a whole was. so any qah rework will not stop people from complaining about bad maps getting ranked. i would argue that stopping bad maps from getting ranked is not a good goal to begin with because aside from pp abuse which messes with competitive integrity of the game's leaderboard, the average "bad" map does not have a major impact on other maps or the game as a whole.

I mean the issue fits in to 1 line:

Noone is doing QAH

The rest of the discussion occurring here is just derivative of the issue and figuring out reasons why to figure out solutions, u can just do a root cause analysis flowchart/workflowy list to organize the shit too

Crissa wrote:

don't really think it can be solved tbh since the way people will see you won't change, you'll always be the bad guy that constantly checks maps looking for mistakes, also it's ultra boring and you gain absolutely nothing

yes thank you it is hardly a secret that QAH is currently a role with net negative returns, the goal here is to either come up with some form of motivating factor that is enough to see success OR, failing that, find some other system entirely to replace the current qualified system because it does not work if no one is there to enforce quality

just did QAH to get a better grasp of what it was like; in roughly 3 hours i managed to check 8 maps, a whole day's worth of ranked.

given i was also the only one in the qah list besides cheri, i'd have to spend 3 hours every single day doing essentially 2 nominations worth in activity, while also having people get mildly tilted at me for "dq modding" their map (if i find anything)

in my view QAH winds up being equivalent in time to just actually modding one of your BN reqs, except you're not modding anything other than severe issues (so it's not very interesting) and its' unlikely to be a set that you like (thus very boring to stare at for half an hour)

i dont really have any solutions to propose, just wanted to give some insight

peppy should pay BNs and the staffs to increase motivation, even by little.

their work don't get recognised at all. At least mappers can get their maps noticed in YT and mentioned by players.

Haachama chama~!!

paying people to do a dogshit boring/unfulfilling task they wouldn't ordinarily want to do is not the solution nor will it ever be the solution, stop suggesting it

In my opinion one part of the problem is that quality assurance is an almost 100% backloaded process at the moment.
In the worst case a map gets submitted, nominated and then qualified while only being checked by 2 people.
At that point the milk has already been spilled from a QA perspective.

A nomination is already some kind of a "seal of approval" greatly encouraging mappers to look for a second BN and at the same time deterring them from looking for more mods. At the same time it may deter modders from modding that map because they may be afraid of wasting their time on a bunch of defensive "this is already nominated so this must be fine" replies to any points they may rise. Ultimately for most mappers going for ranked the goal is to get the map ranked, not to make it as high quality as possible.

The current nomination process encourages looking for BNs instead of looking for mods if your map is even remotely good.
A process that encourages to get to the nomination on the fastest way possible is naturally not bound to produce quality but will instead result in maps getting nominated that barely meet a bottom line.

Endaris wrote:

In my opinion one part of the problem is that quality assurance is an almost 100% backloaded process at the moment.
In the worst case a map gets submitted, nominated and then qualified while only being checked by 2 people.
At that point the milk has already been spilled from a QA perspective.

A nomination is already some kind of a "seal of approval" greatly encouraging mappers to look for a second BN and at the same time deterring them from looking for more mods. At the same time it may deter modders from modding that map because they may be afraid of wasting their time on a bunch of defensive "this is already nominated so this must be fine" replies to any points they may rise. Ultimately for most mappers going for ranked the goal is to get the map ranked, not to make it as high quality as possible.

The current nomination process encourages looking for BNs instead of looking for mods if your map is even remotely good.
A process that encourages to get to the nomination on the fastest way possible is naturally not bound to produce quality but will instead result in maps getting nominated that barely meet a bottom line.

The thing is encouraging people to look for more mods doesn't solve the issue with QAH in the slightest. nothing guarantees those mods a mapper gets before qualified will point out anything useful. the difference between a map having 100 mods and almost none is so small at times that this wouldn't help. there also lies the issue of this not helping with unrankables very much since if the nominating BNs didn't catch them those non BN modders very well might not either.

Ur right Endaris in that people just aim for the lowest bar since that's the only standard that exists in the current timeframe of osu and just go straight to BNs and dont really care about much else, but getting people to look for more mods outside of the ranking process won't really be the solution since like wot P1 said "the difference between a map having 100 mods and almost none is so small at times", i dont think the incentivization of finding more mods will fix this issue

I'm also not even sure if this is related to QA itself. I guess it could be on the basis of higher standard -> less work needed when doing QA work; though I think focusing on raising the minimum standaard will help very little in alleviating the QA workload since the focus of QA work Right Now AFAIK is moreso on finding unrankable issues + issues that are Arguably unrankable and were missed such as spread issues and fucked lower diffs moreso than "raising quality" from the minimum standard; in other words, it could might as well just increase the workload moreso than anything else by raising a higher standard

quality assurance IS all about finding unrankables and obvious mistakes.
there's a veto system for subjective concerns.

I feel like we have historical proof why QAT didn't work. A small group of people deciding the ultimate fate of what gets ranked or not will just be a source of conflict between people. Also the job sucks and nobody wants to do it. Existing QAT members wud just burn out super fast.

Uberzolik wrote:

just did QAH to get a better grasp of what it was like; in roughly 3 hours i managed to check 8 maps, a whole day's worth of ranked.

given i was also the only one in the qah list besides cheri, i'd have to spend 3 hours every single day doing essentially 2 nominations worth in activity, while also having people get mildly tilted at me for "dq modding" their map (if i find anything)

in my view QAH winds up being equivalent in time to just actually modding one of your BN reqs, except you're not modding anything other than severe issues (so it's not very interesting) and its' unlikely to be a set that you like (thus very boring to stare at for half an hour)

my initial thought there is having QAH worth more from a BN activity perspective, but even that doesn't fix the fact that it is both probably going to be a set you don't like and screening purely for severe issues, based off your experience anyway

i'm not really seeing a tenable way to make QAH stuff more attractive to people without overweighting it in terms of offered reward/incentive, especially when the whole "dqs are bad" thing is going to turn anyone who does QAH with any regularity into a social pariah

doing QAH for activity is already overweighted as it is. you can easily farm enough activity to not need to nominate maps but nobody does it because there's no point.

Kibbleru wrote:

I feel like we have historical proof why QAT didn't work. A small group of people deciding the ultimate fate of what gets ranked or not will just be a source of conflict between people. Also, the job sucks and nobody wants to do it. Existing QAT members wud just burn out super fast.

I feel like this sort of ignores the proposed solutions. Both of the suggestions involving a QA group have involved keeping NAT in place. The issue with QAT was that they were, as you said, "A small group of people deciding the ultimate fate of what gets ranked", but the main issue with that was that they were essentially also the top of the chain of command. With either of the proposed solutions involving a QA group (either a separate QA group, or giving people in NAT a job of QA) in theory, NAT would also be on a level playing-field enough to speak out against QA doing stupid shit, and kick the offending members.

In regards to the issue of burnout, I feel like the game growing as much as it is, and just generally being a lot bigger makes it easier to just replace members who lose interest or can't keep up with the amount of work. This also sort of solves the issue of needing to replace people if someone in said group does something stupid.

I'll throw out an idea I thought of off hand, I don't particularly have any strong feelings either way but figure it might inspire an idea in someone else:

What if a QAH like group (could call it QAT, Assistant QAT, 'community modders' or some other title) was created that was relatively easy to get into, like you could have them take a written test similar to a BN and require X number of kudosu (just to weed out the extremely inexperienced ppl who know nothing). Then give them a little badge so they have some incentive and feel good about themselves and let them do QAT, but not have the ability to actually DQ. They could post mods on the thread and a BN or NAT could come along and DQ the map/resolve the mod if it was right/wrong. Or it could be where X number of these "QAH" need to agree on it being an issue before a BN/NAT gets involved. At the very least these "QAH" would be able to find obvious things like accidental cross-screen 1/4. Those who do an exceptional job could get a fast track to a BN promotion, as additional incentive, and those who do poorly could be weeded out.

Basically, BNs don't want to do QAT because it's boring and they could instead spend their time nominating. So what if we used the less experienced modders who can still identify issues to do it for the BNs? A lot of them might be able to identify issues, but maybe not at the same level/consistency that a full BN can. I'm sorta thinking along the lines of BN applicants who got reduced cooldowns or came close to reduced cooldowns, etc.

Anyways, dunno if that idea sucks, but hopefully it inspires some ideas in someone else

-White wrote:

I'll throw out an idea I thought of off hand, I don't particularly have any strong feelings either way but figure it might inspire an idea in someone else:

What if a QAH like group (could call it QAT, Assistant QAT, 'community modders' or some other title) was created that was relatively easy to get into, like you could have them take a written test similar to a BN and require X number of kudosu (just to weed out the extremely inexperienced ppl who know nothing). Then give them a little badge so they have some incentive and feel good about themselves and let them do QAT, but not have the ability to actually DQ. They could post mods on the thread and a BN or NAT could come along and DQ the map/resolve the mod if it was right/wrong. Or it could be where X number of these "QAH" need to agree on it being an issue before a BN/NAT gets involved. At the very least these "QAH" would be able to find obvious things like accidental cross-screen 1/4. Those who do an exceptional job could get a fast track to a BN promotion, as additional incentive, and those who do poorly could be weeded out.

Basically, BNs don't want to do QAT because it's boring and they could instead spend their time nominating. So what if we used the less experienced modders who can still identify issues to do it for the BNs? A lot of them might be able to identify issues, but maybe not at the same level/consistency that a full BN can. I'm sorta thinking along the lines of BN applicants who got reduced cooldowns or came close to reduced cooldowns, etc.

Anyways, dunno if that idea sucks, but hopefully it inspires some ideas in someone else

What BNs/mappers here are apparently believing - according to the replies to my previous post - is that normal modders are too incompetent to spot unrankable issues that slip past BNs which makes their contribution worthless for the purpose of quality assurance.

Regardless of whether you believe in the worth of these modders or not, if there's any chance they will actually spot something, the system should ideally have them check before a nomination/qualification to avoid the social and organisational minefield that the DQ process is right now.

Any osu! player already has the opportunity to leave such comments right now on qualified maps anyway, it's not a particularly inspiring task which is why none is doing it and I highly doubt some shiny badge with no power is going to change anything about it, not to mention the extra effort in supervising and "rating" these people which may as well result in a similar rating disaster as the current DQ point system.

White's idea seems somewhat sound on paper if it's for checking unrankables and not assuring "Quality". A lot of the social minefield in DQs results from them seeming too subjective or unnecessary. I believe DQs for strict unrankables are easier to tolerate despite still being annoying.

To me the main unknown factors in running something like this successfully would be along the lines of
1. does interest match the scope, even with potentially raising the ranked cap in mind
2. user retention or user inflow, it won't mean much if people growing bored leave faster than others join
otherwise everything else I can think of is more or less answerable (?), could be missing something though

1. can be polled
2. is hard to predict without actually trying it - it's hard to say how much these less experienced modders will value their badge

I'd also like to note that this doesn't necessarily guarantee fast tracking into BN, most applicants fail based on modding ability which this system doesn't train or test

-----

@Paul Blart - you're basically saying "users that are trustable with moderation powers are easy to recruit and replace" which unfortunately isn't true.
also fwiw I'm not against the QAT idea because it's "been done" but because the issues with it are known and rather harmful

yaspo wrote:

White's idea seems somewhat sound on paper if it's for checking unrankables and not assuring "Quality". A lot of the social minefield in DQs results from them seeming too subjective or unnecessary. I believe DQs for strict unrankables are easier to tolerate despite still being annoying.

To me the main unknown factors in running something like this successfully would be along the lines of
1. does interest match the scope, even with potentially raising the ranked cap in mind
2. user retention or user inflow, it won't mean much if people growing bored leave faster than others join
otherwise everything else I can think of is more or less answerable (?), could be missing something though

1. can be polled
2. is hard to predict without actually trying it - it's hard to say how much these less experienced modders will value their badge

I'd also like to note that this doesn't necessarily guarantee fast tracking into BN, most applicants fail based on modding ability which this system doesn't train or test

-----

@Paul Blart - you're basically saying "users that are trustable with moderation powers are easy to recruit and replace" which unfortunately isn't true.
also fwiw I'm not against the QAT idea because it's "been done" but because the issues with it are known and rather harmful

If the issues with the system are known wouldn't it be more beneficial to discuss them, and possible ways to prevent them?

Just seems kind of backwards to not even attempt to discuss why the old system failed, and what could be prevent that from happening again (especially since it feels like multiple people in this thread know why it failed). It takes less time to fix a system with a few problems (assuming you know what they are) than it does to create a new one from the ground up, so with a relatively urgent issue like this I don't see how brushing off discussion of fixing an old system is productive at all.

Anyways I don't feel like this will change anyones mind on even attempting to discuss why QAT didn't work and how to fix it, and I don't see any more than surface level discussion of other solutions, so I don't have much more to say with this thread.

You say fix a few problems - the problems were exactly what this thread is trying to discuss. The reason QAT burned out so fast and pushed for a change was basically because they were reduced to unrankable-checking bots (current QAH) and BN evaluators (still going alright overall). Similarly, the problem here ("problem" - I don't think it's actually a problem) is that nobody wants to check unrankables and do the bare minimum expected of two other BNs for them.