https://docs.google.com/document/d/1EbhpZviwcvAYM926IsKZ-Q8DKRH4N3_WdpYcD5JdD-U/edit?usp=sharing <-- post to read

https://twitter.com/Cheri_Osu/status/1369671204580306949 <-- Twitter post

On request by Ephemeral
-
community/forums/topics/1268711?n=14 <-- referring an QA extension/Adding a branch
community/forums/topics/1268711?n=15 ^ Me at the same time basically stated a similar thing in detail in a document

CONS ^ was similar to QAT and tried it before didn't work/don't want to repeat anything and lack of interest still

community/forums/topics/1268711?n=59 my adjustments on -White's post
Basically about making a third group w/no BN benefits for non-bns to do QAH to add incentive instead of relying on BNs who don't want to do it.
community/forums/topics/1268711?n=46 ^ the original

CONS ^ No point in doing this after being BN, Having to trust non-bns is risky even with considering they can't dq, not as reliable as a BN doing it, rewards can be considered iffy,etc.

community/forums/topics/1268711?n=85 <--- abolishing the system

community/forums/topics/1268711?n=126 ^ my tl;dr version why just abolishing it isn't the best idea

community/forums/topics/1268711?n=107 ^ the full version + just be me
explaining the same thing in my document a bit better

community/forums/topics/1268711?n=123 <-- refers to having QA as a group making RC proposals and such
community/forums/topics/1268711?n=130 ^ basically rejecting it as RC is fine as is

community/forums/topics/1268711?n=139 QoL changes could add

Since it's here too this is my response to the suggestion of splitting the BNG and QAT https://gist.github.com/VINXIS/25aeae000261f07ae1bc52f7f2c25043

tl;dr history suggests its a bad idea

A suggestion is a suggestion at the end + what happen in the past doesn't necessarily mean it shouldn't happen again especially since the game is much bigger now than it was back then and there was obvious a lot more mismanagement back that can be easily rectified if we look more into the core problems.

The suggestion at the end was nothing more my (and friend) 2cents on options to do, I imagine there is much more

I agree, we should be looking more into the core problems, I just felt it was necessary to respond to that and that the same thing will happen again because the environmental factors caused are still about the same because of those core problems

everything else in the post I think was good and how there has been so little action by the NAT either especially considering the amount of data that is pretty easily achievable that could be used to solve issues

VINXIS wrote:

everything else in the post I think was good and how there has been so little action by the NAT either especially considering the amount of data that is pretty easily achievable that could be used to solve issues

what sort of data do you mean here, exactly? is this in regards to punishing dq's or something else?

Hi, author of the first half of the google doc here

VINXIS wrote:

tl;dr history suggests splitting the BNG and QAT is a bad idea

I mostly came up with this because despite QAH being a volunteer position within the BNG, there is no real obligation attached to it, so there is no artificial motivation to perform QA as a BN (on top of the no/negative motivation to do so naturally). Splitting off the QAH/QAT/whatever you want to call it was what I envisioned as a rather extreme method of defining the roles of those who are actually willing to do this work (sure, there are probably none that are willing to do it to the extent that Cheri does, but I'm fairly sure there are plenty that would do QA at least to some extent if they knew it was to fix a system on the verge of collapse as things currently stand).

Theoretically, this would also be solvable with QA quotas for ALL BNs, but that would likely deincentivize BN applications at all, so not exactly an ideal solution. A large BNG is good with how big the game has gotten, but not good if the other critical systems in place (such as QA) can't even begin to keep up.

I recognize that the question of how to motivate QA activity is extremely difficult, but especially given the current additional load on the qualified section thanks to the mug event I think it's more important than ever that there is at least some serious discussion on what to do about it.

Some points of data I think would be beneficial off the top of my head

1. Nature of vetoes presented
2. Nature of the disqualifications in general
3. The states of qualified and ranked throughout this period of using such a system
4. Survey analysis

This is in regards to helping find out what may have indirectly/inadvertedly caused what when using the system

aidanbh wrote:

Splitting off the QAH/QAT/whatever you want to call it was what I envisioned as a rather extreme method of defining the roles of those who are actually willing to do this work (sure, there are probably none that are willing to do it to the extent that Cheri does, but I'm fairly sure there are plenty that would do QA at least to some extent if they knew it was to fix a system on the verge of collapse as things currently stand).

That's fair enough, though I do think the only time this type of change will be fruitful is when the treatment of the dq + discussion system is changed

VINXIS wrote:

That's fair enough, though I do think the only time this type of change will be fruitful is when the treatment of the dq + discussion system is changed

agree on this point, will discuss further at some point later today(?)

VINXIS wrote:

Since it's here too this is my response to the suggestion of splitting the BNG and QAT https://gist.github.com/VINXIS/25aeae000261f07ae1bc52f7f2c25043

tl;dr history suggests its a bad idea

You have a pretty fair point but it's important to note that the reasons the QAT was hated back then are rather easy to mitigate; qualified maps no longer give pp (which fueled a lot of player hatred towards disqualification) and often times QAT posts were very subjective and contentious (see: Asymmetry, first DQ is post 107 [did you know you can click the number and go to a specific post? I had no clue until now lol]). The pp debacle was settled a while ago, but I'd also argue that 'subjective DQ' debacle is effectively settled with the introduction of the veto (and mediation) system. As flawed as it is, it is a good replacement for the old "QAT says No thanks" system of before; now, a large portion of the BNG is required to give fairly detailed reasoning as to why a veto should or should not stand. A separate user group could just essentially be the 'janitors', clearing out obvious spread issues, straight-up missed unrankables etc.

I don't necessarily think that a new user group is the best decision, but it's definitely not as apples-to-apples as it might seem on the surface.

For the record, I entirely agree with the idea that the way we view disqualifications is the root cause, and I'd suspect that a lot of that has to do with the time aspect. You can wait days or weeks for a BN to find time for your map which you likely spent dozens of hours making, or you can search for days for BNs to wait for weeks, and by the time that the map is qualified, the absolute last thing you want to do is have a last-minute disqualification throw another wrench into the system. From a mapper's perspective (especially one without close ties to BNs), it isn't as much of mutually-beneficial quality assurance as it is just pure pain. Though honestly, I don't see that there is a fix for this issue without an extreme reform (like, no-BNG level of extreme) that, frankly, I'm not sure how it could possibly be done.

In my perspective at least as a mapper I think that if im given suggestions + stuff and im in a process where the discussions are not meant to be a benefit to the general ranking system, then the process of applying mods in qualified should be so that all i have to do is apply and update, no need for a DQ to apply and update, and Then have to wait for the bns to be free to check and nominate again after many days, and Then on top of that also have a punishment given to the bns just because I myself was receptive to feedback while in the qualified section that I thought would improve the map And were also not even unrankable issues Yo hailie can we get utsuroi qualified again yet, I think that that would at least to some nontrivial degree make it far less painful to be receptive to feedback in qualified for everyone, and also Wayyy back when the 'Fix Vetoes' Thread existed I was suggesting version history to make this far easier to do.

Also unrelated since this is 2 discussions at once, but I definitely think the 'subjective DQ' debacle is nowhere near effectively settled using the veto + mediation system because it is effectively the same process with the facade of a form of discussion occurring in the veto state and in the mediation state, which is why I was saying that the suggestion in the current environment of things would definitely not be good at all. I do agree that it is better than the shit QAT used to do for sure and removing pp and queue time reset has significantly helped, but I don't think that is enough for us to consider it environmentally different enough still because the root issues from then are still in effect today

nobody wants to do QAH work quite simply because its fucking boring and there's no incentive. you're not even doing an actual bn check you're just checking for unrankables and giving it a general pass of "yeah i guess its fine mapping wise"

speaking as someone who did a boatload of them in the past and will never do one again, I wish I could say something beyond "theyre boring" but it's just the facts. the fact that you still do them is honestly incredible to me (in a positive sense)

given that I don't see why anyone would actually want to be a QAT in your proposed system. its just being a BN but without any fun. and given that the compensation for the work will inevitably be nothing (as usual) nobody would want to do it lol

---

Also, given that mapping is a community sphere, I dislike how QAH work makes you look like an asshole if you dq any maps. people like to say "well the maps shoulda been good" and yeah you're literally right but we also live in a world with human beings that have emotions. nobody likes getting their map dq'd. being a QAHer puts you in a position where you're directly contributing to punishing your fellow BNs.

hard agree with how stupid and inconsistent the "severity" rating thing is

In response to hobbes:

giving some sort of incentive to do QA is an issue, and I do agree a lot of people would consider it boring, but I feel like that neglects the fact that some people enjoy things that are considered mundane or tedious to most. Since the community is growing so rapidly there are more people who would be willing to do said tedious tasks.

In regards to the comunity backlash aspect I feel reletivly the same way. Bigger comunity means more people who are able to not care about backlash.

----------------------------------------------------------------------------------------------

In general I feel like being less fixated on the singular proposed would be really beneficial for this entire ordeal since, at the end of the day it's just that. a suggestion from a singular person. It's sort of counterproductive to have such a strong unmovable stance in this sort of thread in my opinion since qualified is being held together by a single thread that's on the verge of snapping. New ideas, or ways to improve existing one's are the most important thing here.

----------------------------------------------------------------------------------------------

In regards to the proposed solution in the doc:

While I don't know if bringing back what is basically old QAT will be sustainable in the long term I feel like it's still worth discussing, even as a temporary fix while a more long term solution is worked on, especially since it doesn't seem like it would be overly hard dev side (may be wrong on this, so correct me if I'm wrong).

In my opinion old QAT would have worked if, one there was enough people who were willing to do the boring work and didn't care about community backlash, and two if there was some sort of way to moderate stupid decisions made by the QAT members.
I feel like the proposed solution of 3 teams (BN, NAT, QAT) could very well solve that issue. Some sort of objection function on the mapper's side that leads to discussion between NAT and the non offending members of the QAT, which could result in the offending party getting kicked if their DQ was too out of line. Having that out side pair of eye's (being the NAT) could potentially aliviate some to a lot of the bias in whether the DQ in question was out of line, and the appropriate punishment for the offending QAT member.

I also feel like the main reason why a lot the reason people who would otherwise be willing to check qualified don't is because being on a level playing-field with the people you disagree with is hard, so giving those people who would be willing to a role with similar power, or perceived power as NAT would make them more likely to check qualified.

Obviously this is just my two cents on the issue, but I strongly feel like something needs to be put in place to deal with qualified sooner than later, so going in circles on whether one idea will work really won't help.

well the alternatives are:
- slowing the entire ranking process WAY down in order to thoroughly check every map in qualified before letting it through. this would ensure quality, but would make a lot of people very angry as way fewer maps would be getting ranked and it would be tiring for those doing the checking. obviously not sustainable with the current scale of the game.
- ditch the QAH system entirely and just let stuff go through qualified (this is functionally what we have right now because as you said no one wants to do QA voluntarily). also not sustainable since as the game grows, so will the rate of major issues getting through qualified undetected.

so one of two things needs to happen:
- we find a group of people who, against all the classical theories of motivation, are willing and able to dedicate themselves almost exclusively to QAH work, and set them aside to do that specifically. I think Cheri proves that they exist, even if they are few and far between. it's also possible that a group of people exists who would not do it in a vacuum, but would do it with even a tiny bit of external motivation (such as the positional things mentioned in the document/by P1T)
- thoroughly rework some other part of the system to reduce the number of unrankables getting qualified in the first place. I know I said in the document that it was impossible to entirely eliminate human error in the BNG, but it may be possible, by raising the standards of BN performance, to get it down to an acceptable level such that QAH becomes unnecessary

honestly my take on this is -

issue is - right now qah isnt incentivised at all

also within the NAT we have quite a few people specialising in different things (think pishi & naxess compared to what noffy & seto do)

so what could happen to solve all this is have sub divisions to the NAT group (not seperate titles just offical different obligations within NAT) to include QAH work, so you have 3-4 NATs that their primary job is to do qah, 3-4 NATs that their primary job is to do new and current BN evals etc. (not delegating this among current NATs but promoting a few qah centric modders to fill these roles)

this would solve the issue of giving an incentive to do qah (getting nat leadership position) while also getting a more active and consistent userbase doing qah

i mean rn the qat probably dont want this change to be made - but its either this, make it mandatory for bns (which everyone will hate) or just not do qah at all (which seems counter-productive with the influx of new bns over the last two years or so)

i fully expect no changes to be made but with the current state the entirety of qah basically rides off of cheri doing the work - so what happens when she stops doing it?


also this would avoid trying to split up the bng which was tried before (think tier 1 & tier 2 bns etc)

https://docs.google.com/document/d/1BfOpVJt05aNobf0zoDTOFzgVyFbC7kzAPB4aj--QvGU/edit?usp=sharing Deviouspanda basically stated similar to what I was going for with this document but nonetheless i'm adding it here anyways since I was told to

All and all I simply just want to be able to not be the only one doing qualified and having solutions like above ^ would be a good thing even if temporarily to solved that problem until more reworks that will take much longer than the simple fix here.

Im not sure how it is for anyone else here but I think if I was in the position to, I would personally be more than down to do QA work or post mods on almost all maps in the qualified section myself if

1. it only focused on the aspect of improving the map and discussing with the mapper
2. it didn't have to affect anyone outside of the realm of the mapper and myself

which is why I wanted to suggest removing the dq punishment system and getting version history and everything instead of trying to "force" people to get into QA work, but idk how many more people feel similarly on working on QA so maybe making more people to QA work in the NAT or similar really would be the best option, idk tho

VINXIS wrote:

Im not sure how it is for anyone else here but I think if I was in the position to, I would personally be more than down to do QA work or post mods on almost all maps in the qualified section myself if

1. it only focused on the aspect of improving the map and discussing with the mapper
2. it didn't have to affect anyone outside of the realm of the mapper and myself

which is why I wanted to suggest removing the dq punishment system and getting version history and everything instead of trying to "force" people to get into QA work, but idk how many more people feel similarly on working on QA so maybe this really would be the best option, idk tho

I'm not denying that having such could be potentially helpful in the dear future as honestly I wouldn't even have to go through qualify the way I do if things was more laxed as such, but that is a system that takes work and still as well as still gotta considered that qa itself is a rather boring task on it's own.

My solution is basically to create management for the very core needed part of the game with the influx of mappers and nominators. So whilst yes a perfect world, going with solutions like this would be cool, but right now I think having something more QOL up until things are more stabled, should be prioritized.

fair enough, I just don't have much trust nor expectations for quick changes anymore because of how the past 14 years of osu have ONLY been just quick changes (and not just in the ranking system, in EVERYTHING related to osu) but it could be worth for now i guess


Edit: though at the very least the suggestion for raising QA activity value shuold really be done as well honestly

VINXIS wrote:

Im not sure how it is for anyone else here but I think if I was in the position to, I would personally be more than down to do QA work or post mods on almost all maps in the qualified section myself if

1. it only focused on the aspect of improving the map and discussing with the mapper
2. it didn't have to affect anyone outside of the realm of the mapper and myself

which is why I wanted to suggest removing the dq punishment system and getting version history and everything instead of trying to "force" people to get into QA work, but idk how many more people feel similarly on working on QA so maybe making more people to QA work in the NAT or similar really would be the best option, idk tho

Honestly I don't think getting rid of DQ punishment would be a bad idea, hell even just making it more lenient and consistent would help.

I feel like if there was quality control of some sort in place there would also be less need (excluding more extreme examples such as BNs who have a lot of generally agreed on questionable noms, but thats sort of case by case).

Some sort of QA being in place would likely encourage BNs to be a bit more careful in general so I feel like it very well might work.

Also having more people checking qualified would definitely make focusing more on improving the map and having discussion when it comes to subjective issues a lot more viable.

I feel like a lot of the issues (and you can correct me if im wrong here) people have with the way cheri DQs stem from the lack of time and support given to her to check qualified. Having more people doing the job would give more time to discuss, and provide more meaningful feedback/mods.

Checking qualified alone basically means to be able to get through everything and do more than check for unrankables DQing more immediately if the mapper doesn't want to change anything is more efficient, but given more time I'd assume she would prefer not to do that.

Yea i was talking about removing the DQ punishment/sev rating in the posts above since the focus for DQ punishments should really only be for bad actors, and i dont think someone modding a bunch of qualified maps improving the beatmaps and getting them changed shouldnt be considered as the BNs that nominated those beatmaps being bad actors for example, would even help with the current mods that cheri is doing in QA currently