Hi everyone, in light of the many unranks/mistakes that slipped to the Ranked section during the past couple of years, the NAT has been working hard for months on a way to fix and incentivize quality assurance, and we're glad to present to you a brand new team: Qualified Inspectors!

We basically took the core concept of the previous quality assurance system and modified 2 main aspects:

1. QA will now be centered over objective issues (such as unrankables), rather than subjective issues
2. QA is no longer a BN-exclusive task

Very solid idea, look forward to seeing how this is implemented in the months going forward.

Excited to see how this turns out!!

Damn good and hope it can be real in other modes

looks nice!

Solid idea on paper. Good way to incentivize QA'ing which has always proved to be lackluster!

cool

[Proposal/Concern]

Qualified Inspectors are made-up of experts in any given gamemode, who can regularly check Qualified maps and propose fixes. Great, solid idea, no qualms there, it's the same as the Beatmap Nominators.

However, a common weakness for a majority of modders, is nuances such as Metadata, Audio, or non-mode specific additions to a beatmap.

This doesn't seem to be addressed though, unless I'm mistaken. My concern is, that even if someone were to show great contribution to these areas and show expertise above even the current Quality Inspectors who are checking these areas too, it won't be rewarded in the same way. Or, they could be an expert in a mode and one of these nuances, but are now restricted to that mode, otherwise their work is meaningless, which is what this group seems to eliminate in the first place.

Has this been discussed or how will this concern be alleviated?

less powerful QAT is what this is? Cool for those who don't want to try for BN but want to help out in the mapping scene. Will definitely try this out and see how it goes when it becomes public

Question(s) though, will QI be a way for people to get into BN or will it exclusively be QI centric.

If a BN wants to step down into QI, how would that work?

Also, do they get more priority than those wanting to join?

Thanks for NAT leadership for doing change and implementing ways to help osu! progress.

seems good, as long as we're 100% making sure that they're only DQing for objective issues. everyone should hopefully still remember all the problems with the QAT (the subjective dqs being a god awful mess) and im glad to see this being different in a good way

while i still don't think this group is necessary and unranks/mistakes are already sufficiently minimized and properly addressed (assuming higher-ups don't randomly intervene), this is a nice incentive for more people to contribute to qualified and, if nothing bad comes out of it, would be a net positive

also the name "qi" sucks just do qat 2 electric boogaloo

edit:

aceticke wrote:

However, a common weakness for a majority of modders, is nuances such as Metadata, Audio, or non-mode specific additions to a beatmap.

my honest opinion is that this is being done as a direct response to a recent incident that involved a major unrankable but was not unranked due to internal disagreements, and this is a way to get around those disagreements. thus, only major gameplay issues are being focused on, with these nuances also generally not being too important in the grand scheme of things

aceticke wrote:

[Proposal/Concern]

Qualified Inspectors are made-up of experts in any given gamemode, who can regularly check Qualified maps and propose fixes. Great, solid idea, no qualms there, it's the same as the Beatmap Nominators.

However, a common weakness for a majority of modders, is nuances such as Metadata, Audio, or non-mode specific additions to a beatmap.

This doesn't seem to be addressed though, unless I'm mistaken. My concern is, that even if someone were to show great contribution to these areas and show expertise above even the current Quality Inspectors who are checking these areas too, it won't be rewarded in the same way. Or, they could be an expert in a mode and one of these nuances, but are now restricted to that mode, otherwise their work is meaningless, which is what this group seems to eliminate in the first place.

Has this been discussed or how will this concern be alleviated?

If we expect QIs to check all of these nuances, the only thing left for the BNs are subjective changes, which are getting lesser and lesser over time, particularly for already established mappers. The goal isn't to do a 2nd BN checks, and these details are part of a normal BN check already. Additionally, maps do not really get unranked because of meta or audio issues, I think the main goal of QI is to ensure a map is functionally rankable, not completely polish every corner (as that is a BNs role, again).

Oh I do have a couple questions:

1 - What is the timeframe for sets to be able to be unranked? Score loss should still be mitigated - and say for example it's not the same to unrank a set minutes after it goes live than a week after. A certain timeframe should be given.

2 - Can a set be unranked if a rule change occurs after a beatmapset goes live? I strongly believe there should be a clarification that a set should be judged based on the rules that existed before it was Ranked. It would be unfair to apply changes retroactively unless they were objectively disruptive to the experience.

That's all! Just thought these two thingies should be brought up earlier rather than later.

agree with nifty

if you wouldn’t dq it immediately or if it requires extensive discussion before a dq, then it shouldn’t be part of a QI’s job but rather a BN’s

re: spread concerns, *some* meta issues, structural issues that don’t completely destroy the map, break issues to an extent, fringe cases for audio + compressor differences

here’s the tldr, if u expand QI too much then it’ll become a mess like QAT was. Less is more and the current objective-focused version may just be what the qualified section needs

Good luck getting BNs to actively want to do nuance checks in Qualified though. And if there is a glaring meta issue then it should be brought up by the QIs if noticed, not just put aside as it's not essential to the map's functionality.

a good idea!

i play taiko

I think it needs to be specified what counts as objective issues and what doesn't, is it limited to things that break rules and are therefore unrankable or does it also include important guidelines that are clearly broken with no apparent justification? If so, can maps be disqualified immediately for such issues? There are many cases where guidelines are not followed which have a major impact on gameplay and are generally considered objective problems. But there are also guidelines where people disagree on whether it's a problem or not.

As a side note, I feel like checking for subjective issues that affect the map's quality is also important, isn't that the goal of QA after all? Obviously it should not be done to the same extent as the QAT in the beginning (i. e. enforcing preferences and disqualifying maps based on that, essentially a hard veto) but I think Qualified Inspectors should report these issues to BNs/NAT which would then decide if they're valid (QI should not be able to DQ maps themselves for these things). I believe this would also give QI more purpose and incentives.

by objective issues we mean certain unrankables, a.k.a rule breaks.

How are we able to assign ourselves to maps? The QA page is still read-only for people not in the BNG or NAT.

Hivie wrote:

Notice: this system is currently in a closed trial phase for the osu!taiko game mode

after the closed trial ends, the system will be open for everyone, with access to the QA page being public

This looks promising, hopefully it goes in the right direction!

Finally, I can DQ over incorrect metadata/clipped audio!

why only taiko for the (closed) trial phase though?

rHO wrote:

why only taiko for the (closed) trial phase though?

Yea, that's a great question, std could get a few too for sure

I think that's because osu!taiko is one of the simpler mode to play, so it's easy to look at objective issues on these maps...

rHO wrote:

why only taiko for the (closed) trial phase though?

because taiko strikes a perfect balance in community size and map amount to test these kind of experiments. other modes will be included relatively soon once things feel more stable.

Is QI also involved in the pattern aspect? Or does it only deal with rc part?

Murumoo wrote:

Is QI also involved in the pattern aspect? Or does it only deal with rc part?

it only deals with RC, or more specifically rules where the issues pointed out are objective. we cannot dq any maps due to subjective issues (patterning/visuals or anything that is subjective) unless they fundamentally break rules such as unsnaps etc.

I essentially did QI checks on random Qualified maps before, so it's good that this is a thing now.

Conclusion

After more internal discussion among the NAT + members of the osu! team, we have unfortunately decided to scrap the idea of QI for now, and possibly re-visit it in the future.

however, we'll be directing more attention to our current Quality Assurance system to try and improve the situation a bit, so we've made the following changes:

1. QA will now be more oriented towards objective issues: while BNs are still free to post about subjective stuff, they are only expected to check each diff for objective unrankables, primarily gameplay-related one (checking metadata only is not a QA check for example)
2. we will be offering some nomination activity leniency depending on your QA performance on a case by case basis, up to a minimum of 6 noms in a 3mo eval cycle. we would like to emphasize that we do not want this to be a primary way for BNs to dodge minimum activity, this is meant to be as an exception and a reward for people who have outstanding QA performance, but didn't reach the bn activity reqs. so don't treat this as an expectation nor take it for granted.
3. QA performance will be considered when selecting elite nominators: this doesn't mean it will be a requirement for elite nominator, this only means that we'll offer a spot for people with good overall performance + outstanding QA activity, so BNs who excel at other areas shouldn't be affected at all.

over the coming months, we'll be working on better ways to streamline QA activity which would allow us to possibly have more rewards in the future.

the NAT would love to hear your feedback about QI and other ways to improve/incentivize Quality Assurance, so please let us know of any concerns/feedback/ideas that you might have in this thread \o/

Hivie wrote:

Conlcusion

[list] QA will now be more oriented towards objective issues: while BNs are still free to post about subjective stuff, they are only expected to check each diff for objective unrankables, primarily gameplay-related one (checking metadata only is not a QA check for example)

Does this mean that BN applications are still going to have the subjective aspect of it? If BNs are now only regulated for objective issues does the Ranking Criteria portion mean more than the "submit mods" portion?

Blushing wrote:

Does this mean that BN applications are still going to have the subjective aspect of it? If BNs are now only regulated for objective issues does the Ranking Criteria portion mean more than the "submit mods" portion?

the way we evaluate bns will not change, QA is when BNs check qualified maps for any issues, and that's the only part affected with this change.

Hivie wrote:

Conlcusion

After more internal discussion among the NAT + members of the osu! team, we have unfortunately decided to scrap the idea of QI for now, and possibly re-visit it in the future.

however, we'll be directing more attention to our current Quality Assurance system to try and improve the situation a bit, so we've made the following changes:

1. QA will now be more oriented towards objective issues: while BNs are still free to post about subjective stuff, they are only expected to check each diff for objective unrankables, primarily gameplay-related one (checking metadata only is not a QA check for example)
2. we will be offering some nomination activity leniency depending on your QA performance on a case by case basis, up to a minimum of 6 noms in a 3mo eval cycle. we would like to emphasize that we do not want this to be a primary way for BNs to dodge minimum activity, this is meant to be as an exception and a reward for people who have outstanding QA performance, but didn't reach the bn activity reqs. so don't treat this as an expectation nor take it for granted.
3. QA performance will be considered when selecting elite nominators: this doesn't mean it will be a requirement for elite nominator, this only means that we'll offer a spot for people with good overall performance + outstanding QA activity, so BNs who excel at other areas shouldn't be affected at all.

over the coming months, we'll be working on better ways to streamline QA activity which would allow us to possibly have more rewards in the future.

the NAT would love to hear your feedback about QI and other ways to improve/incentivize Quality Assurance, so please let us know of any concerns/feedback/ideas that you might have in this thread \o/

Why was it decided to scrap QI altogether? What was the problem?

I feel like these changes barely make a difference - most BNs will still have no time, motivation and/or incentive to regularly check qualified maps.

That's why I still think non-BN users should also be part of Quality Assurance and it should be extended to important non-objective issues as well (see my post above).

Rewaring QAs is quite difficult to make it "fair" across all modes if we think about objectivity, since each modes have its own scope of rules that vastly differs in term of checking difficulty. The amount of incentives should be also considered, especially if the previous idea also includes mappers outside of BNG to assure the quality.

I'm curious as to what specific conclusions were made that influenced these decisions. On paper this seemed like an at least fair approach to the issue surrounding the qualified section and it's ever growing quality concerns that, If refined a bit would make QA a bit lest daunting for BNs alike as the workflow would be more evenly split. It's a bit hard to give really a proper diagnosis on what exactly could be changed/improved without some amount of background knowledge.

Having QI be purely based on objective issues is certainly a good move making things less frustrating/ambiguous from bother parties of which being the mapper/BN, As far as incentives are concerned I don't really have a say on that. QA by nature probably might not be as appealing as being a BN by which is seen as a far more prestigious role to take given QA's bad reputation in the past as well being less "hands on" in a way. I'm not also even sure if newer modders who wish to strive for BN or want to become more involved in the modding side of the side of the community are even aware this side of the bubble, not entirely at least until much further down the line as not much light is really shed on this.

For me personally, If there's to be a specific group of people working on QA it really needs to feel like its own thing with its own set of perks separate (sort of) from BNs with more laid out rules & expectations. In an ideal case BNs and QIs work together yet separately to ensure maps are of the highest quality while being able communicate among one another especially in cases where things might get a bit fuzzy with what could be seen as an objective or subjective issue. Again, I have no idea what the census was as to what exactly went wrong with this so there is little room to really give in-depth insight.

I like the idea of having modders who might be fairly experienced in what they do but might still lack some skills to elevate themselves to BN status or simply not having the time for it but want to want to get more involved in the ranking process could be offered an additional alternative at least as something to do on the side. Another common concern is BN themselves not having the time to do these QA checks on map, The incentive to lower nom threshold seems good but where do you draw line with what is considered an outstanding QA performance and not. Is there specific amount of successful QI checks that need to meet? Is it relative among each other or is it entirely subjective? This is something that doesn't really need to be hard-locked behind a quota that is already a hassle on it's on in my opinion.

Unfortunate. This would have given new mappers an aim while they are not skilled enough to get into BN.

Serizawa Haruki wrote:

Hivie wrote:

Conlcusion

After more internal discussion among the NAT + members of the osu! team, we have unfortunately decided to scrap the idea of QI for now, and possibly re-visit it in the future.

however, we'll be directing more attention to our current Quality Assurance system to try and improve the situation a bit, so we've made the following changes:

1. QA will now be more oriented towards objective issues: while BNs are still free to post about subjective stuff, they are only expected to check each diff for objective unrankables, primarily gameplay-related one (checking metadata only is not a QA check for example)
2. we will be offering some nomination activity leniency depending on your QA performance on a case by case basis, up to a minimum of 6 noms in a 3mo eval cycle. we would like to emphasize that we do not want this to be a primary way for BNs to dodge minimum activity, this is meant to be as an exception and a reward for people who have outstanding QA performance, but didn't reach the bn activity reqs. so don't treat this as an expectation nor take it for granted.
3. QA performance will be considered when selecting elite nominators: this doesn't mean it will be a requirement for elite nominator, this only means that we'll offer a spot for people with good overall performance + outstanding QA activity, so BNs who excel at other areas shouldn't be affected at all.

over the coming months, we'll be working on better ways to streamline QA activity which would allow us to possibly have more rewards in the future.

the NAT would love to hear your feedback about QI and other ways to improve/incentivize Quality Assurance, so please let us know of any concerns/feedback/ideas that you might have in this thread \o/

Why was it decided to scrap QI altogether? What was the problem?

I feel like these changes barely make a difference - most BNs will still have no time, motivation and/or incentive to regularly check qualified maps.

That's why I still think non-BN users should also be part of Quality Assurance and it should be extended to important non-objective issues as well (see my post above).

sounds like modding v1 all over again,they should just bring it back at this point

Hivie wrote:

After more internal discussion among the NAT + members of the osu! team, we have unfortunately decided to scrap the idea of QI for now

why? what was brought up in the mentioned discussion?

I don't even want QI to return, just curious. I feel like this should be fairly obvious but explaining publicly the various objectives, benefits, failures, unexpected side-effects, etc., would help everyone better understand the situation