Probably has been said in this thread before but badges and supporter are great incentives for most people, see the beatmap spotlight lobby and mappers guild (which would be way less popular IMO if there was no badge incentive). Starting with perma supporter for BN is honestly a pretty good change.

More badge incentives, not sure for what exactly, would also help motivate people for tasks such as QA, higher activity, whatever else

Also maybe the elite nominator tag being given more frequently (like every year maybe?), cus it would motivate people to perform better as a BN

elite is supposed to be yearly and a big fat new batch is coming in the next few days with community contributor

Ephemeral wrote:

elite is supposed to be yearly and a big fat new batch is coming in the next few days with community contributor

Cool! didn't realize it was supposed to be yearly since I don't believe we got elite nominators last year. Thanks for the correction!

Excited to see who else gets put up as elite nominator :eyes:

wafer wrote:

Ephemeral wrote:

elite is supposed to be yearly and a big fat new batch is coming in the next few days with community contributor

Cool! didn't realize it was supposed to be yearly since I don't believe we got elite nominators last year. Thanks for the correction!

Excited to see who else gets put up as elite nominator :eyes:

I know it's been a long year but our first batch of EN were literally this time last year soo ya

Sylvarus wrote:

To get back on topic, I'd be interested to know people's reason why they think QAH should exist in the first place and if those goals can even be realistically achieved in any way.

In my mind removing QAH won't be much different from the current state.

The whole document and it's idea was under the assumption that we had to work with either literally what we got or only minor dev things

Considering the whole turn of events I am more interested in seeing if there is a different way to go about this that can be more effective and can be easier to agree upon than the previous discussion at the time, which wasn't really going anywhere.


EDIT: I am still on the fenced that having some form of QA and a mix between having more refined evaluations to increased bn quality over all is still necessary, it just a matter of which would be a more preferable approached

Because at the end even with BN quality standards changing, there will always be a need of an eye on qualified regardless unless it gets to a point where any issue thereafter can be done through a bot in which case that'll need to be discussed a bit more here

the number of qualified maps allowed per day has just increased to 10, on a vaguely related note

DeviousPanda wrote:

...but at some point you need to stop relying 100% on the community to run the mapping scene for you when we desperately have been asking for changes and fixes for a while now - fixes that we cant implement as the community ourselves (and this is what nao is reffering to when disconnecting core team and community)

if you want an example of why this disconnect is real - just look at the bn server - the activity and discussion of the dev team is literally non-existent there (and that is the primary place where bns organise themselves)

Any web changes that need to be made can be made by someone in the community who knows how to code web frontend/backend. Bounties are offered to people willing to help on the dev side. I dunno if by "fixes that we cant implement as the community ourselves" you thought someone from community can't implement changes, or if you meant that you don't think there is anybody in community with coding skills for it.

K after recollecting my thoughts and with the new situation at hands i'm going to lay out what i'm thinking about and being a bit more open-minded.

The issue currently is that we still do not have a good sustainable way of making sure we aren't too lenient on quality if we want to do away with things such as QAH

As it stands, a dq gives a lot more information and how much of an issue is relevant towards a BN more than just looking at a map that has been rank and telling them the issue after the fact.

If you are too lenient, we have the case of where we have right now, the more bns we get, the more work it is too evaluate them and any amount of feedback can be just as difficult and straining than it just looking at qualified since you have to see a massive load of maps to look over if they are doing something properly even if you just go by only 5-10 out of whatever how many they nommed.

That also not to mention that it'll be basically turn into "QA" without dqing if you think about it, which is uh arguably just as boring and more counterproductive.

And with the continuous load of bns, that too will be just as much of a hassle than it is to keep up with qualification without increasing the size of the NAT as a whole.

With no QA, evaluating bns in nature would be lenient as long as there isn’t a set standard to be place and how you guys are going to judge off of quality issues and that on it’s own needs to be discuss as the bng as a group have way too many varied opinions to have a set and stone answer for this.

And being too strict can easily cause stress on those who is generally already doing well which in general is creating something way too ideal of what it means to be a BN and still needs a set and stone standards.

A world without any QA, sounds good on paper but in reality would not work just as much as the current system relying on random dqs/bubbles and a sole person checking everything at this moment to set some form of punishment and strictness in evals and nobody can argue it ain't too lenient as of right now.

My point is regardless of what many may see, you always gonna need to have something more active to pay attention to qualified and potentially questionable things to go off regardless of what standards are put in place for evaluations.

Either that or you have the case of putting too much on NATs which let’s be honest now, their workload is going to only get bigger unless they are even more strict on things such as BN apps as the time goes on

I’m not saying QAH/QAT is needed as in like a group, but i’m just putting my thoughts on why just having no QA in general would lead to just as much of a havoc than it would good (which didn’t conclude bns getting outrage, etc.)

Tl;dr - if it won’t be just as bad as the current QAH situation now, it will later.


Now that I have that out of the way, it is clear that you still would need some form of QA management no matter how you go about this, even if it is a small group of BNs being the driving force of it all. Either that or now that we have more potential of dev support, implement something that could make this a bit more sustainable in the long run

Every single conclusion still leads off to “who will even do such?”

The core issue is still with the amount of backlash you get from even just vetoing a map regardless if justified or even just posting a suggestion on a map, that’ll always draw a line between people wanting to help out in this field, no matter the rewards

It still a great thing to push down some rewards even when considering that, as even if the rewards is not enough to keep people from doing it long, there is still the chance of those who dedicated their work to helping, should still be notice on this

Elite nominator badge and all that aside, a title + badge still wouldn’t hurt anyone regardless and I think that on it’s own can be easily implemented eventually. This isn’t really to draw attention to it but rather to just simply have it there because quite frankly, it should have been there to begin with.

The other thing is people simply burn out easily from it.

^ That is an issue that’ll never be changed unless it was made mandatory in general which isn’t really pleasant for anyone out there.

And I have no general idea how we would go about it elsewise outside of simply taking the “Q” in the title more seriously.
Regardless of whether or not the original purpose, the original purpose at large, never played that big of a part back even when this was created as unrankables was still easily missed but other quality issues were more noticed and simply put was more enjoyable to look for than being a QA bot.

Quality control is a vague thing to only go off of as we all have our standards of quality hence the veto system and mediation system.

And with those in place, people still won’t just go out of their way to just veto anything no matter how much authority they get, if it doesn’t mean they have enough support and they can easily get whatever since of power rebuked or at the very least, the mapper only lose out on a week in qualified and the map is back and going if it doesn’t get upheld.


Even for example we did make a QAH team or w/e QAT knock-off, these people will still have the consequences of being punish for doing anything stupid or too nitpicky unlike in old times and this should never be an issue to do such if they do anything wrong.

The only thing that is preventing any form of QA management being truly a good idea is just simply put, current BN’s interest and the incentive to actually maintain it which still involves needing a big enough of an gain for the BNs in question, to at least always have 1-3 people available for the specific job.

And having a minimum of 3 people maintaining it (and hey have a say in BN evals) would create more of a middle ground than too far into the left or right.

Anything else such as possibly could cause conflict such as lowering standards does not mattor too much here as other systems as cons that could be great be worse than that issue here and as long as there's a middle ground, I highly doubt that the standards can be lower than they already are in this current state.

Whether we need to create interest by having a third group or not does not matter, what matters is that we still need it and that is my point.


Sorry if that is a lot to read i'm just simply trying to make sure that we have a balance system.

I agree with Cheri that doing away with QA wouldn't be a good idea. If we do I would argue we should require every map get 3 BNs at least for rank since 2 BNs clearly aren't enough to reliably catch every issue, but I don't think that's a very good idea either.

-White's idea for the "Quasi-QAT" doesn't really seem that bad. If we remove the "stepping stone" mechanic for BNs from the original concept, there are plenty of modders who can, and are willing to mod for unrankables, assuming this is the only job this "Quasi-QAT" would have. And these modders who most likely would be a bunch of no-name modders would be pleased to receive any kind of reward or recognition for their help which BNs wouldn't normally perceive as anything incentivizing.

Why not just run with -White's idea for now, along with Cheri's adjustments? I say even throw in a supporter tag for QA members, and tie it to the minimum required activity per month or something. Monetary incentive is the best incentive.

What about long term issues? Do you really think there's a long term solution without flaws? I don't. The system could work for a year or two, really depends on the amount of new people wiling to join in. Doing something is better than doing nothing. I'm sure you will need to have this discussion every few years and decide on changes, until something sticks that is.


And concerning Nao Tomori's point about identifying unrankables vs adding value to a map. It may be that constantly looking at qualified maps might develop a quality standard in a person. And you know, people who mod maps in qualified don't only mod for unrankables, not all of them at least. There's bound to be a large portion of people interested in being a part of the QAT just for the badge or whatever benefits they will have, however that doesn't render everyone else obsolete. Even with just a small percentage, nomination team could potentially gain more members at a steady rate.

That's my side of the coin anyway.

edit: deleted last bit in fear of being retarded, but also cuz the paragraph was redundant

3 BNs would be a terrible idea. It would make ranking maps an INFINITELY harder chore, especially more niche maps that only a few bns may like.

would also reduce content load by a LOT... def better solutions to quality issues, though I honestly take a stance similar to Sylvarus where I believe QAing as a position/role is uneeded.

From what I'm seeing in this thread, it seems like the biggest issue is that BNs are missing quality issues that make a map inherently unrankable, and the systems in place for handling this problem aren't doing a good job. Does that seem about right?

In a perfect world, a program would exist which decides in finite time whether a given map has anything inherently unrankable. In this perfect world, a map would be REQUIRED to pass the check done by the program before a BN could be asked to bubble. This may not be possible mathematically (given a reasonable amount of computing space/memory - discretization means the set of rankable maps is finite and therefore, with enough space, one can write a program which checks exhaustively) with the GIVEN criteria, but certainly the criteria could be changed so that a program CAN decide in finite time whether or not a map meets it. The most obvious way of doing this is for all NECESSARY criteria for a map to be UNrankable are rigorously, objectively defined, but this isn't necessarily the only solution.

Many criteria are caught by AIMod already. Can someone with more modding experience list the criteria for unrankability that seem hard to make a program capable of recognizing? One obvious one is "diff spike in low star diffs of a spread" not being rigorously defined; another is, potentially, "incorrect metadata." Both of these can be patched (if imperfectly, but we don't reach perfection even if humans were doing QAH reliably) by having more objective versions that cover the same general cases. As an example, for the first, we could change it to: "for a map of x stars, the maximum difficulty is at most f(x)", where f(x) is some function derived by the community. Do we lose something here as compared to the original criterion? (in other words, are there maps that would satisfy this criterion that would not satisfy the original, and vice versa?) Certainly. The results of automated QA should be able to be disputed with a BN (perhaps it could cost kudosu to dispute automated QA), and a BN should still be allowed to cite the more general criteria when choosing not to bubble a map, but having this level of rigor and automation in the QA system would reduce the amount of maps needing DQ to practically zero, would it not?

Is there a reason this hasn't been proposed before?

JPK314 wrote:

Is there a reason this hasn't been proposed before?

Probably because there isn't anyone who could or would program such an AI?

wafer wrote:

3 BNs would be a terrible idea. It would make ranking maps an INFINITELY harder chore, especially more niche maps that only a few bns may like.

There's a reason I said that it was a bad idea, but the point is that 2 BNs are not enough to catch unrankables, so unless we have QA as a safety net, something else will need to be done. An AI program (imo) isn't feasible given that a lot of disqualifications are for inherently subjective things anyways (look at Super Driver for an example of that. Nothing inherently unrankable there, and I'm pretty sure there were 3+ BNs willing to nominate it).

M i X wrote:

I say even throw in a supporter tag for QA members, and tie it to the minimum required activity per month or something. Monetary incentive is the best incentive.

This is not the solution, and your statement about monetary incentive is the best incentive is just factually wrong. Money is in no way the best incentive, and the only sustainable way to motivate people is to provide intrinsic motivators rather than extrinsic. Doing that is difficult, but I definitely am strongly against offering supporter for those participating in my suggestion.

@-White Supporter only when a certain quota is reached, or maybe for top contributors. Intrinsic motivation can only get you so far.

-White wrote:

JPK314 wrote:

Is there a reason this hasn't been proposed before?

Probably because there isn't anyone who could or would program such an AI?

This is definitely not the reason. An AI in the machine learning sense is NOT what I'm talking about - the bigger part of the suggestion is applying slight modifications to the more ambiguous unrankable criteria so that it IS easy to code a program to check it automatically.

community/forums/topics/1268711?n=107 to keep relevance and waiting for a response for

I do like to say that those who keep saying it's should be abolish and whatnot, please understand that this is basically the one thing that I truly cared about more than anyone else and that I much rather be the only one doing it than getting rid of it entirely.

Quality Control was something I cared about ever since I was a bn doing qah when it was first made and I always been striving to care for it more

The whole entire thread people has been nothing but dismissive of the point and completely forgotten what this thread was even about.

Please understand the intent and that removing it is not helping me in the slightest, and if anything just remove my ability to show how much I care about my cause since other methods (like evaling i.g) is not optional and it gives me even less of an influence than I already have right now which is completely wrong.

So with that said, please let's be more productive instead of denying every single possibility to improve it.

In terms of Quality Control I'd personally probably prefer a different approach than checking every single damn mapset on a qualitative basis.

I think we'd gain a lot more mileage by digging down (Root Cause Analysis type beat) and figuring out what's really behind these common quality issues, and then looking for a broader approach? Broader being stuff like knowledge sharing, stimulating actually modding, looking into adjusting how we deliver content to different target audiences, etc; anything that can meaningfully change the environment rather than having to fight against the current. It's less controlling but ends up being a more positive effort.

Ya 100000% agree that itd best thing to do for figuring out how to solve the Quality Control issue(s) weve got going on with QA and BN nominations and all

But you stated has not in any form has address any of my issues as well as just being a utopia that can easily fall just as flat.

"In terms of Quality Control I'd personally probably prefer a different approach than checking every single damn mapset on a qualitative basis.: But completely diminishing an approach that one has been doing quite for years and not having any solid solution is completely wrong on your part.

If I wasn't ignored almost 99% of the thread, I would love to actually think of more than just one solution here but nobody has produce a single one that even relates to the topic or can be considered sustainable and that is what I do not get.

All it has been like I said dismissing one idea and not giving another one that actually can be agreed on as a whole.

doing a Root Cause Analysis flowchart or some form of workflow/process diagram in listing these issues would definitely help organize everything regardless of the feasibility of that tho

and it also looks like its more of a necessity for this as each day passes honestly, would also help in figuring out which suggestion/recommendation to try out first

yaspo wrote:

In terms of Quality Control I'd personally probably prefer a different approach than checking every single damn mapset on a qualitative basis.

I think we'd gain a lot more mileage by digging down (Root Cause Analysis type beat) and figuring out what's really behind these common quality issues, and then looking for a broader approach? Broader being stuff like knowledge sharing, stimulating actually modding, looking into adjusting how we deliver content to different target audiences, etc; anything that can meaningfully change the environment rather than having to fight against the current. It's less controlling but ends up being a more positive effort.

I feel like on paper not needing QA is a good idea, but that leads down the rabbit hole of "how do evals work if there are no DQs" and "what determines if a BN should get probation because of their noms".

While it would be nice to be able to say "well, there will still be enough DQs to judge BNs come eval time" I feel like thinking in terms of how things ideally will turn out never goes well in regards to things like this. You always have to consider the "what if", and how that can be dealt with. To combat that without a proper QA system some other form of judging BNs for evals would need to be considered that doesn't involve DQs and sev ratings.

The only thing I can think of would be judging BNs based on their ranked noms (since we would have to assume next to no maps are getting DQed), and that would essentially lead to NAT having to do QA on ranked maps in order to eval BNs which just sounds like more work from the get-go. Basing evals on ranked noms would also have other issues, such as BNs having no immediate feedback to messing up. This could very well lead to good BNs being careless because they have no indication that they missed a spread issue or a contrast issue on a low diff for instance, and then when eval time comes they will have no clue why they got probation/negative feedback. While I do think that positive reinforcement is better than negative I don't think they can exist without each other, which is why I don't think this would work.

Obviously there might be other ways to handle evals without DQs that I can't think of, but I have a feeling most, if not all, of them would be a lot of work and be very difficult to manage.

The other issue with removing QA is that if there is some magic solution that solves all the issues it will still take a lot of time and discussion, and it would be best to have something in the mean time even if removing QA is the end goal.

I feel like it would be beneficial to at least attempt something along the lines of a QA group, or at least poll for potential interest in it, considering it seems like the easiest solution to attempt and get rid of/iterate on if need be.

VINXIS wrote:

doing a Root Cause Analysis flowchart or some form of workflow/process diagram in listing these issues would definitely help organize everything regardless of the feasibility of that tho

and it also looks like its more of a necessity for this as each day passes honestly, would also help in figuring out which suggestion/recommendation to try out first

Also agree here, would really help us stop running in circles.

peppy wrote:

we will continue to support any changes that you guys see beneficial (and roll them back if they turn out not to be).

apparently we have freedom to change the ranking system as we see fit so if no meaningful changes happen it's only because the people managing the ranking system don't want to or are scared to do any meaningful change.

some concerns BNs and other users seem to have in common:

1. beatmaps ARE getting qualified with unrankable issues or obvious mistakes (because BNs are humans and make mistakes too)
2. but doing QA means hurting fellow nominators ​and "becoming the vilain".
3. because DQ's are bad.


the solution seems to be as simple as getting rid of punishment so mappers and nominators become more receptive to DQs, Nominators will be more willing to do QA, DQs will no longer be bad news and will be considered just an inconvenience at most.

edit:
the NAT should be helping BNs that are not performing well instead of punishing them, stop turning everything into paperwork and hiding behind it.

just my thoughts

first of all i see most people here talk like QA is just about finding unrankables like unsnaps, audio issues, etc

i believe that QA, as the name implies, should be about assuring the quality of the maps that shall enter rank, not check if they have basic unrankables. also nobody gets mad at a dq about 320kbps audio or an unsnap because it's just a stupid mistake that happens. and a dedicated QA group isn't needed for that matter. most issues like these already get treated in qf without an actual group dedicated to them

the moments where you can face backlash is when you'll go and dq a map for an actual quality concern (low diffs being too hard, content bloat, just a map being low quality...) and that's what the goal of a QA team/group whatever should be. and i think most will agree that there is too many maps that could've used a dq like this but ended up getting ranked with no problem

the problem is nobody wants to do that job if there isn't anything in return. if there's something in return people will do the minimum (only dq for dumb unrankables) to get the reward


tbh only thing that can be done without big dev support or feeding people with rewards to force them into doing qah is upping the standards. make BN evals more about the quality of the maps they nominate and less about their clean nom / dq ratio. punish people that push low-quality content and try to evaluate BN applicants' actual mapping standards

when the whole pp abuse drama with horrible kids and start again happened, i believe the quality of new qfed maps went up (for a short period of time). why? because BNs saw 5 others get kicked for pushing lower quality maps

there is no need for something like this to happen again but i'm sure the number of "bad" maps in qualify will decrease just by letting BNs know that yes, the quality of the maps they nominate will now matter

I propose that we allow the QAH group to add/change the RC every few months or at set points in the year. It can work like congress where members propose rules and if a majority pass it, it gets in. This system would help solve most of the quality control issues as the QAH group would be made of members who are very experienced in the issues that get pass the current RC. It would also provide a better incentive for people to try and join the QAH group and get what they want from the RC.

Currently BNs get punished lightly if a mistake is not an obvious or glaring issue from what I understand based on VINXIS' statements. However issues like snap and things on the RC are punished more heavily. If we go through with this change, we can remove BNs that aren't checking the map as they are suppose to and can hopefully limit the amount of QA needed in the future. (cause all the bad stuff would already be outlined in rc)

(^if that isn't already the case, we should punish BNs who rank maps that go against the RC harder, its a list of things to check, just go down the list and check off 1 thing at a time)

"DQS are bad" We can limit the arguments about subjective issues by using the RC. Any judgments made on subjective issues can be added to the RC by the QAH group and going forward, people can propose new rules to make all DQs objective based on the RC.

Bot wrote:

I propose that we allow the QAH group to add/change the RC every few months or at set points in the year. It can work like congress where members propose rules and if a majority pass it, it gets in. This system would help solve most of the quality control issues as the QAH group would be made of members who are very experienced in the issues that get pass the current RC. It would also provide a better incentive for people to try and join the QAH group and get what they want from the RC.

Currently BNs get punished lightly if a mistake is not an obvious or glaring issue from what I understand based on VINXIS' statements. However issues like snap and things on the RC are punished more heavily. If we go through with this change, we can remove BNs that aren't checking the map as they are suppose to and can hopefully limit the amount of QA needed in the future. (cause all the bad stuff would already be outlined in rc)

(^if that isn't already the case, we should punish BNs who rank maps that go against the RC harder, its a list of things to check, just go down the list and check off 1 thing at a time)

"DQS are bad" We can limit the arguments about subjective issues by using the RC. Any judgments made on subjective issues can be added to the RC by the QAH group and going forward, people can propose new rules to make all DQs objective based on the RC.

the issues that causes dqs other than what's already in the RC is subjective stuff that needs to be evaluated from case to case, there isn't a practical way to implement those issues in the RC

Only relying on evals/removing QA tl;dr so this stop being ignored on why this can cause a problem.

1. What are you evaluating off of? There are too many BNs with a variety of opinions to set a standard without going through convoluted loops, which would cause leniency in nature and/or be unclear from the outside come eval time, making BNs more mad because of the lack of immediate feedback that could be given by a DQ.

2. There is NOT enough NAT to support the continuous growth BNG on their own. Even increasing the standards to get in isn't really going to cut it, as time and time again that has shown it has its own problems.

3. It will just end up being like QAH without the DQ part majority of the nominations that you are evaluating.

^ How are you going to work around not checking people's nomination if you want to set a standard that everyone can follow? You still need to make sure things are in order and it just simply going to be a huge load on yourselves.

4. Giving feedback on how to do better on your quality standards is not enough and causes influx of work with the amount of BNs while effectively not stopping much of any problem because we do not live in an ideal world where everyone is willing to change, and it will just become similar to QAT, except without DQs.


Can we at least look at why QAH won't work as a Third Group/QA management branch in more detail than looking for an idealistic path that simply put won't go too far without having something far more complicated to deal with and potentially more boring than what we have right now?

Why would you want to add more work for yourselves as a small team, when you can split it up into 2 parts instead?

Why can't there be both, having more feedback on evaluations whilst also having more immediate action to prevent questionable stuff from getting rank in the meanwhile?

Why can't we test this out as now that dev support is more easier to get before concluding that this won't work?

I ask this because the arguments I had against my idea has been flat and manpower truly is not gonna be an issue if we just do a poll on it and go from there

Since right now I feel like it is better to at least attempt to test something that can make this easier for us all, than to argue in circles why something won't work here at all as I truly wish I gotten a better argument about it than what I have gotten thus far, or i'll just continue to push this idea here since I do in fact want something more immediate.

Either that or wait for solutions that can be more realistic in practice.

Kudosu wrote:

Bot wrote:

I propose that we allow the QAH group to add/change the RC every few months or at set points in the year. It can work like congress where members propose rules and if a majority pass it, it gets in. This system would help solve most of the quality control issues as the QAH group would be made of members who are very experienced in the issues that get pass the current RC. It would also provide a better incentive for people to try and join the QAH group and get what they want from the RC.

Currently BNs get punished lightly if a mistake is not an obvious or glaring issue from what I understand based on VINXIS' statements. However issues like snap and things on the RC are punished more heavily. If we go through with this change, we can remove BNs that aren't checking the map as they are suppose to and can hopefully limit the amount of QA needed in the future. (cause all the bad stuff would already be outlined in rc)

(^if that isn't already the case, we should punish BNs who rank maps that go against the RC harder, its a list of things to check, just go down the list and check off 1 thing at a time)

"DQS are bad" We can limit the arguments about subjective issues by using the RC. Any judgments made on subjective issues can be added to the RC by the QAH group and going forward, people can propose new rules to make all DQs objective based on the RC.

the issues that causes dqs other than what's already in the RC is subjective stuff that needs to be evaluated from case to case, there isn't a practical way to implement those issues in the RC

We can just do polls between BNs/Nat/whoever within a certain timeframe of a subjective issue, and go with majority rules on subjective topics, then we can add that decision on rc

QAH cant work without incentive. No one wants to be be the dude who DQs maps, its extra work and you dont get any benefit for it. Who is going to volunteer to do extra work for no benefit? (besides making the rank section better ig)

obviously we cant award monetary rewards so im suggesting we give the qah the power to change the rc (again, helps promote better quality maps and people now know what to look for better)

JPK314 wrote:

-White wrote:

JPK314 wrote:

Is there a reason this hasn't been proposed before?

Probably because there isn't anyone who could or would program such an AI?

This is definitely not the reason. An AI in the machine learning sense is NOT what I'm talking about - the bigger part of the suggestion is applying slight modifications to the more ambiguous unrankable criteria so that it IS easy to code a program to check it automatically.

"Unrankables" doesn't always refer to stuff in ranking criteria. It can also refer to subjective stuff like a slider not facing the correct direction to be create a more symmetrical pattern, a map being too bland, wrong hitsound usage based on song, or whatever other thing it might be. You basically need something that guesses what imperfections in abstract material might be and suggests how to make it better. That's a no easy feat.

Bot wrote:

the power to change the rc

Everyone has that power already, making a second, completely separate way to change the RC is just counterproductive. Discussing potential changes on the forums is infinitely better than any sort of system that's limited to a privileged group.

the RC proposal process is in a good state as-is, don't think that needs to be touched at all

abraker wrote:

JPK314 wrote:

-White wrote:

JPK314 wrote:

Is there a reason this hasn't been proposed before?

Probably because there isn't anyone who could or would program such an AI?

This is definitely not the reason. An AI in the machine learning sense is NOT what I'm talking about - the bigger part of the suggestion is applying slight modifications to the more ambiguous unrankable criteria so that it IS easy to code a program to check it automatically.

"Unrankables" doesn't always refer to stuff in ranking criteria. It can also refer to subjective stuff like a slider not facing the correct direction to be create a more symmetrical pattern, a map being too bland, wrong hitsound usage based on song, or whatever other thing it might be. You basically need something that guesses what imperfections in abstract material might be and suggests how to make it better. That's a no easy feat.

Isn't that what BNs are for, though?? I'm not saying we remove BNs, just that we remove the need for DQs by increasing the standard that reaches BNs in the first place to beyond anything DQable. Are you really saying that subjective issues that not all BNs agree upon (demonstrated by the fact that at least two BNs approve) are worth DQing a mapset over?

yes

Yes absolutely. There are subjective DQs all the time. A major reason the BNG (as I understand it) is as big as it is is to allow for lots of perspectives to be represented. Some BNs think that anything that meets the RC should get ranked, and others have subjective quality standards they think that ranked maps should meet, for one example.

So.

Over this weekend I'm gonna read through this thread and make a new post or something summarizing the major ideas + pros / cons + issues people are seeing. We can use that to help with getting back on topic and a bit more focused.

I wonder if a format similar to how let's rework qat stuff! handled multiple proposals would work? A major issue I'm seeing here is people are discussing multiple different proposals at once which makes it harder to follow and respond to in depth.

Cheri, what do you think?

Considering that the thread only has this

community/forums/topics/1268711?n=14 <-- referring an QA extension/Adding a branch
community/forums/topics/1268711?n=15 ^ Me at the same time basically stated a similar thing in detail in a document

CONS ^ was similar to QAT and tried it before didn't work/don't want to repeat anything and lack of interest still

community/forums/topics/1268711?n=59 my adjustments on -White's post
Basically about making a third group w/no BN benefits for non-bns to do QAH to add incentive instead of relying on BNs who don't want to do it.
community/forums/topics/1268711?n=46 ^ the original

CONS ^ No point in doing this after being BN, Having to trust non-bns is risky even with considering they can't dq, not has reliable as an BN, rewards can be considered iffy,etc.

community/forums/topics/1268711?n=85 <--- abolishing the system

community/forums/topics/1268711?n=126 ^ my tl;dr version why just abolishing it isn't the best idea

community/forums/topics/1268711?n=107 ^ the full version + just be me
explaining the same thing in my document a bit better

community/forums/topics/1268711?n=123 <-- refers to having QA as a group making RC proposal and such
community/forums/topics/1268711?n=130 ^ basically rejecting it as RC is fine as is


Outside of that there hasn't been much solutions as far as I see (Vinxis first post don't count as store history is already being added) and anything else that I can add here is redundant (I summarize the cons on the first 2 but it was just skimmed)

tl;dr there is basically only 3 solutions on this thread that is relevant to the topic

Well, that's a start, thank you. I'll still be looking back through everything though in case anything was missed.

we don't need QAH, at least not as a 'group'. anything that goes beyond checking for unrankables should not be considered QAH work. ensuring quality under subjective standards was a policy that was removed long ago by the staff, you can even go back to 2016 threads and you would find that the old QAT didn't actively check for issues. instead they reacted to reports from the community. this is a decision that was made by the higher ups. they effectively removed the authority to decide what fits in the ranking section from the QAT. it was decided that subjective quality control should be done by the community instead of a dedicated group of 'more capable' modders. whether this decision was a good one or not is something up for debate but that's just how it is.

now having considered that, it makes no sense to have a group of people checking for these type of issues considering that the community was already given the tools to do quality control. things like BNs being able to disqualify maps or reports notifying BNs were made for this. the people that form the QAH are no different to a regular BN so if we were considering QAH work as subjectively agreeing with the maps they mark as fine it would be no different to a 3rd nomination.

i do believe that the QAH is *somewhat* useful in its current state though. having modders checking for issues on qualified maps certainly reduces the amount of maps that are inherently unrankable hiting ranked though if we're getting an excesive amount of maps with unrankable issues into qualified then it's a problem of BNs not being proficient enough, BNs getting too confident and nominating maps without checking them or them not having the right tools to avoid these issues. we could for example have something like mapset verifier run on the osu web when pressing the nominate button so that it minimizes obvious mistakes, have some sort of version control etc.

i think that in the end, if we're considering checking for unrankables only then it's a problem that should be tackled from a different perspective. if you really want to have a group checking for more than unrankable issues then it's something that needs to be supported by the staff. enforcing quality in a subjective manner is not easy especially considering how easily a mod can be rejected these days. mappers and modders are aware of this and they know that continously denying a mod will eventually lead to the modder giving up on the issue and move on since they're just wasting time. if a group of people were to do a job like this they would need to have some sort of authority to deny maps from getting ranked (this has already been done in the past and deemed as not satisfactory).

i think the process of doing QA is similar to evaluating BN candidates (download the map, identify mistakes, report them).
the biggest difference is that you spend more time with the evaluation maps but you check fewer maps, while doing QA you check more maps and spend less time with them.

in reality you will not find issues on many maps during QA checks. it's still a job that needs to be done because the maps with issues do reach qualified.
it's not only a a few members of BN/NAT that are missing these issues, there's definitely those who manage to be more consistent at performing clean nominations but there are just so many things that can be overlooked... everyone miss something unrankable or obvious eventually.

like i mentioned before, i think doing evals is similar to doing QA but why are BNs doing evals and not QA? because there's an incentive to do evals and this incentive doesn't exist for QA(+ everyone hates DQs)...

for that reason i believe the ideas from the proposal could work indeed as long as it focus on what QA means currently (checking for unrankables and obvious mistakes)
instead of just being a new generation of QAT (performing all DQs, being the only people responsable for vetoes etc)
i think that every member of BN NAT should keep their ability to DQ maps and perform vetoes instead of moving this responsability to the new team/nat.
while this new team/nat would focus on what QA means currently, they would still be able to perform vetoes but they would be performing them individually and not as a team.

I edited original post with the only proposals (that was relevant to the topic) on this thread.
There is no reason for a new thread as there was barely an actual discussion to begin with + this is now pinned.
-------------------------------------------
Now on topic

I do like to see a little adjustment on the BN site on how QA checks are being done

Right now on the site
https://i.imgur.com/e7y8sFz.png


^
This is pretty lacking as it is currently and I think some additional tools would be useful



---


Generally a lot of above is nothing more than QoL changes as well as it can help make QA a bit more functional at least on the BN site and more pleasing.

Also this goes to show how much I do think that QA should be considered as a NAT branch, being that this is rather more direct vs indirect feedback from evaluations and can easily split the work if done correctly. Either that or the third group.

---

I think the biggest thing in this thread is that my intent isn't really perceived the way I was trying to go about this and I think the biggest thing is that people are looking at this is if I am asking for QAT back which is rather not fully true in the sense that people is questioning it for.

So with that, I will actually explain my case so we stop the argument "it just QAT and because we did it before, we shouldn't do it again."

Rather than enforcing quality standards upon maps and such, what I'm looking for is simply "assured" the BN on their nominations, and simply give more active feedback in a more frequent matter while checking for unrankables/might as well be unrankable (spreads i.g).

The thing is whilst stuff like probation exist, it isn't stable or what I would considered a consistent way of handling quality over all likewise with giving feedback during evaluation period since it's 3 months after the fact.

One of the things that usually do more outlasting impact is dqs that changes the perception of what is okay or not and apply punishment during these times. But these outcomes such as the incident that got a lot of BNs kick, is that it is rather harsh which is not really a bad thing, but the harshness only implements a scared temporarily as most know things get slowly back to normal over time if that harshness/strictness is not continuously pushed.

That is why along with making the QA page more functional with stuff that should of been there in the first place, I believe it could be more useful and actually brings more of a positive rather than a straight negative.

Having an active QA giving a "review" on a map and their nominators mod instead of just straight up looking for issues can easily change the perception and actually creates a more useful feature to looking at the qualified section.


Because right now as MV and all that is getting more and more useful and possibly we will have things such as a bot to moderate unrankable issues, it can be easily seen why QA is seen rather a boring task even though it is basically similar to evaluations especially with the inherited negative affect it gives from the start since the focus is on "DQs" and such.

Even changing the idea of making it more subjective focus with vetos, can create problems with people as nobody likes to have a DQ even if it is useful to the map, due to a lot of factors with how the system is naturally with the main one being the backlash.

So by the idea of a review, I think simply having the comment part become basically a way to give feedback to the nominator whether it be positive or negative would be highly useful.

Obviously the main focus of QA is still for unrankables, but giving way for QA to tell the nominators they are doing a good job and how well their mods is on the map would give more positive assurance to the nominators and make it less of a nervous wreck when their nominations gets DQ'd at times.

^ Likewise with focusing on criticizing how unpolished the mapset is and how the nominator could be potentially slacking without necessarily dqing the nomination, sets a different tone than a straight up veto/putting the suggestions in mapset itself at the time.

Also this straight up makes it less boring in a sense of just focusing on unrankables, but also doesn't scared people from having a say since it isn't as dq focus as it is currently.

Considering we do not want this to be completely subjective focus, having it more as a way to self-reflect is better in this case and to also keep them on the right track so they could work on their nominations in the future and having DQs that is subjective taken more seriously when it is actually serious or slowly get to that when the negative backlash is lessen over time as nominators/mappers would see the intent is less about pushing/enforcing standards.

Basically leave subjective suggestions to the BNs in hand while QA still follow the job of just making sure things are in order and to help guide the BNs from trouble in the future.

By doing all of above, I think this way of giving active feedback on nominations is a natural positive incentive for nominators alike as it won't apply pressure to them, but instead just keep them from falling short.

The issue with this though is that, It is more work on the QA part to do so.

If QA was an active branch in NAT or Group of it's own, then they can easily focus on this without nominations/evaluations on their own/etc. As well as it just simply gives a better tone for a person who has a title and is meant for this vs a BN as the intent can easily be misunderstood.

Keep in mind that just like evaluation, QA doesn't have to give a review on every single map in qualified as that is insane, but they should still check a healthy amount of maps regardless for unrankables/other issues still.

With a group of 3-4 people and having at least 2 reviews required per member a week and checking at least 4 maps for other issues, would be a nice balance currently.

Over all, I do think this would actually create a positive incentive naturally on it's own and remove some of the negative features of doing QA such as it being bland.

While it is true it will change the intent of QAH but that literally doesn't matter because as long as it is an boring incentive even if it is useful, people will continuously ignored it especially being it also have a negative side to it with backlash.

And changing it where it gives less work for the NAT (as if QA is considered basically something to work together with them), would also create less negative vibes as a whole if it is more improvement focus vs punishment focus and only push punishment when the nominators simply still rejects feeback/still careless.

I hope with all of that I've said, my intent is more clear to anyone who read this thread and with that I am done updating this post (Except for fixing grammar issues.)

Has any further discussion happened in regards to any changes being made moving forward? Or have any conclusions been made?

The way things are now seem very unsustainable, unfair and unfun with a lot of things in the current system causing a lot of stress for nominators. I feel that some change to fix the issues outlined in cheri's doc and other issues in the bn system (ask them not me) would be very healthy for the whole mapping community which would then also benefit players. At the very least, some sort of clear verdict or a further response from NAT would help to elaborate the official stance on these topics and the goals the current systems are trying to achieve.

Lots of good ideas have been thrown around and Cheri did a great job summarizing the relevant proposals. So are we going to see any direct action?

Its really disappointing to see so little personal initiative by most of the NAT members considering Cheri has been carrying QA on her back for so long now. Regardless of the outcome, there needs to be more effort to discuss these issues effectively, and come to some sort of conclusion.

Just to add a little on this aswell:
I think the idea of QAH in general is good. Having people check for unrankables in the qualified section before maps hit ranked should definitely still be done, but differently. I can agree with pretty much everything that cheri has stated towards the QAH and with the community just growing and more mappers ranking maps, it's definitely needed to have some form of quality assurance, since that is what qualified is for in the first place and having a dq happen (which will probably happen more often once newer mappers rank their maps) is much better than having to unrank a map.
With the suggestions that cheri made here I think it's important to adress them, because they have a really valid point that shouldn't be overlooked.

it takes time to formulate results for these kinds of proposals (especially in this sort of volume). the issue is being discussed though and i'm sure there will be movement on the matter eventually, though i doubt it'll be a 100% pull from any of the proposals floated in here, since that's generally not how things work.

that being said, this thread is a super mess. would be nice to get some bullet point summaries of what the issues are and any proposed solutions since i imagine i'm not alone with my head swimming at the prospect of having to pick through this 140+ response thread and several gdocs to see where stuff is at

Cheri wrote:

Considering that the thread only has this

community/forums/topics/1268711?n=14 <-- referring an QA extension/Adding a branch
community/forums/topics/1268711?n=15 ^ Me at the same time basically stated a similar thing in detail in a document

CONS ^ was similar to QAT and tried it before didn't work/don't want to repeat anything and lack of interest still

community/forums/topics/1268711?n=59 my adjustments on -White's post
Basically about making a third group w/no BN benefits for non-bns to do QAH to add incentive instead of relying on BNs who don't want to do it.
community/forums/topics/1268711?n=46 ^ the original

CONS ^ No point in doing this after being BN, Having to trust non-bns is risky even with considering they can't dq, not has reliable as an BN, rewards can be considered iffy,etc.

community/forums/topics/1268711?n=85 <--- abolishing the system

community/forums/topics/1268711?n=126 ^ my tl;dr version why just abolishing it isn't the best idea

community/forums/topics/1268711?n=107 ^ the full version + just be me
explaining the same thing in my document a bit better

community/forums/topics/1268711?n=123 <-- refers to having QA as a group making RC proposal and such
community/forums/topics/1268711?n=130 ^ basically rejecting it as RC is fine as is


Outside of that there hasn't been much solutions as far as I see (Vinxis first post don't count as store history is already being added) and anything else that I can add here is redundant (I summarize the cons on the first 2 but it was just skimmed)

tl;dr there is basically only 3 solutions on this thread that is relevant to the topic

@Ephemeral im p sure that's what this was for, lists proposed solutions and their pros/cons

Ephemeral wrote:

it takes time to formulate results for these kinds of proposals (especially in this sort of volume). the issue is being discussed though and i'm sure there will be movement on the matter eventually, though i doubt it'll be a 100% pull from any of the proposals floated in here, since that's generally not how things work.

that being said, this thread is a super mess. would be nice to get some bullet point summaries of what the issues are and any proposed solutions since i imagine i'm not alone with my head swimming at the prospect of having to pick through this 140+ response thread and several gdocs to see where stuff is at

I made a post if u scroll above as a summary of it and put it on the original post 4 days ago. I didn't add anything that was just redundant or didn't really add much more than what those currently stated.



While I get that they wouldn't straight up take the proposals here 100% and while I understand it takes time,
BUT considering that there is very little of actual discussion around this thread in general along with the fact there is only 4 proposals related to this thread thus far is highly ridiculous to not be talking more in this thread to discuss for further things publicly.

And it just not 4 proposals, it also that 2/3 of them is very likely not going to happen as it is too faulty from the get go (one is literally about abolishing the system and the other was hard denied so uh .-.)

A response is kind of needed to see what they would go about things or if they would in general at the very least soon being that for all I know, they can go an entirely different route due to it being basically only 1 proposal without too many cons to go off of to do a change...

I agree with cheri here, there wasn't much actual discussion or proposals made in this thread (as evidenced by the summary she made), so there's not much to work with. There were really only 3 actual proposals made, and one of them is flat out removing the system (so there isn't anything to pull from there). While I do think under normal circumstances pulling from other people's proposals/ideas is the best way to go about things, like you said, I really don't think that's an option as of right now, which leaves is with two options:

- More indepth discussion on really the only remotely viable proposal

or

- Some other viable proposal to be made within a short timefarme (which isn't very likely)

Frankly, the latter should have been done much, much earlier into this thread, but people were too caught up whining about the origional "proposal" which really wasn't even a proposal at that point. If I have to be blunt here, basically all of the "discussion" done by NAT in this thread was extremely unhelpful and quite unprofessional. The least they could do at this point is discuss the, essentially, singular proposal we have.

Hi, just a short update from my end of things since I get the silence can be a little deafening.

A few of us have been bombarded with questions the past couple of weeks about whether we are going to do anything. Please be rest assured that we haven't forgotten about the concerns brought up in this thread. While we're still unsure of a solid plan to go forward on, most, if not all, of the NAT agree that some changes are necessary.

As it is right now, we're unlikely to take any of the proposals outlined in this thread as is, however that doesn't mean we're going to dismiss them completely. There's currently some plans behind the scenes to put together alternative solutions, and while I can't give an ETA, we aim to get everyone's input on it as soon as it is reasonably possible to do so.

All I'm asking is to just sit tight for a little bit longer.

-Mo- wrote:

Hi, just a short update from my end of things since I get the silence can be a little deafening.

A few of us have been bombarded with questions the past couple of weeks about whether we are going to do anything. Please be rest assured that we haven't forgotten about the concerns brought up in this thread. While we're still unsure of a solid plan to go forward on, most, if not all, of the NAT agree that some changes are necessary.

As it is right now, we're unlikely to take any of the proposals outlined in this thread as is, however that doesn't mean we're going to dismiss them completely. There's currently some plans behind the scenes to put together alternative solutions, and while I can't give an ETA, we aim to get everyone's input on it as soon as it is reasonably possible to do so.

All I'm asking is to just sit tight for a little bit longer.

While I understand that things takes time and that you all need to gather your own thoughts to make a post to get everyone's input and whatnot, the thought of literally ignoring my own statements around this should be talk more and discuss more in general is unreasonable to me especially considering as far as I am concern, I am not even sure if these "alternative" solutions would actually even solved the case and I hate the idea of waiting a while to give input around that.

and by discussion more around, I mean something reasonable instead of how u guys having it internally should be put in place like using the #modding server so you have input as you are making alternative solutions rather than waiting until the last minute or something close resemblance.

We literally had these problems for months so I am expecting more than just a "wait more" response here.

-Mo- wrote:

All I'm asking is to just sit tight for a little bit longer.

Not to be impatient or anything but I feel this is a pretty major issue and there's been some decent suggestions made in the thread that haven't been addressed properly at all. We all agree that changes are necessary, so some kind of update might be nice after 3 months.

The lack of a sense of urgency is disappointing, at least to me. Even if the suggestions proposed in the thread aren't perfect, they're probably better than the current implementation (or at least worth trying) until a better solution can be implemented.

-White wrote:

-Mo- wrote:

All I'm asking is to just sit tight for a little bit longer.

Not to be impatient or anything but I feel this is a pretty major issue and there's been some decent suggestions made in the thread that haven't been addressed properly at all. We all agree that changes are necessary, so some kind of update might be nice after 3 months.

The lack of a sense of urgency is disappointing, at least to me. Even if the suggestions proposed in the thread aren't perfect, they're probably better than the current implementation (or at least worth trying) until a better solution can be implemented.

6 months and still sitting tight eh ?