True af unironically since its kinda interesting as to how receptive bns were to such an idea

Kudosu wrote:

what about a mug?

the event mugs are literally being funded by me and seto kousuke (funds from the bn roast stream are all going to that for starters) and maybe ephemeral (he offered more help after it got started) and would be really another thing on nao's list about things other community members are doing instead of actual from the top support.

i assume novelty would wear off as a permanent item hence different designs for each event to help counteract that (if future events are held)

half the reason i had that idea was because getting bns at least supporter is a thing that for some reason hasn't happened and can't seem to happen despite osu!direct enabling them to do their work more easily among other things

Noffy wrote:

Kudosu wrote:

what about a mug?

the event mugs are literally being funded by me and seto kousuke (funds from the bn roast stream are all going to that for starters) and maybe ephemeral (he offered more help after it got started) and would be really another thing on nao's list about things other community members are doing instead of actual from the top support.

tbh even then most ppl would probably just dq maps for unrankables like unsnaps or metadata. there really isn't many people that want to go through the hassle of dqing for quality concerns when you'll almost systematically face mappers backlash, even if rewarded. nobody feels like its their task

if only there were a way to enfore standards on bns (and without people getting mad at how elitist the ranked section becomes)

@Noffy

Whilst it is true that the point is to reduce the amount of work that a singular person to do Quality assurance, but amidst all of that the biggest thing people are failing to acknowledge within this thread is a singular person who is doing all of this checking within the system is a bn

Even you were acknowledge by QAT/NAT at the time with your ability to be promoted to a higher status, but yet the only thing I ever gotten was being criticized and pure hate even when the vetos/mods were justified whilst still not even considered at all fit for a higher position quite literal no reason at this point and many would agree on such.
(My memory may be fading as I have not remember full details on how you were fully promoted since it been a couple years but my point still stands regardless of such.)

Having things such as meme evaluations stating something similar along the lines “oh wow you did so much QA” basically in the amidst of all of this just felt belittling, and like the work I'm doing wasn't worth anything to you.

If it weren't for all of above ^, then I wouldn't even complain about being the only one (although I still make this thread to improve the system obviously because I do care about it).

Now leaving my personal issue aside, I am fully aware that this is a problem that needs to be discussed more internally above ^ but being that I have yet to even got that opportunity (dear lord I tried.) then placing in the thread is all I can do here.

EDITED: Arguably I do still wish to not have this conversation go any further on the thread since it's personal and I just simply wanted to speak out on it and also do not want to go off-topic even more when I am trying to find a solution (as it quite already is going off-topic too much).

so if you have anything against what I said please take it to dms.


Now as far as the proposal that was presented, I am aware it still has issues and such once they get BN, they won’t do QA anymore, but i’m trying to remain hopeful that it could succeed into having a continuous amount of people willing to come forward and once again, we won’t know if that would actually happen as stated unless tested (which obviously needs dev support,etc)

Ephemeral wrote:

Nao Tomori wrote:

@Eph:
You (staff) are the ones responsible for incentivizing BNs to do the bare minimum. BNs are already "compelled" by way of evals to do that - with fairly good results, honestly, judging by the amount of recent unranks (0) - but we have seen that being more punitive does not improve BN work. Instead, as I mentioned earlier, basic incentives like *SUPPORTER FOR GOOD, ACTIVE BNS FOR EXAMPLE* would go a long way in further ensuring BNs put more care into their nominations and activity. You say nobody wants to put things in place - what more can volunteers put in place? Volunteers have already contributed the following to the mapping community:
- tools to check almost every unrankable
- a gigantic site to organize the official release of licensed content
- a similarly large site to support NAT administrative work like BN evals and QAH activity
- tools to hitsound maps
- tools to create exotic sliders
- tools to create amazing storyboards, and plenty more.
All of these things should have been provided or at least supported by you, the dev team, as part of the development of the infrastructure of the game. Instead, we got the amazing and renowned, very useful, and tremendously inventive HYPE SYSTEM!!!! woo hoo!, we got forced to use the new site which removed several aspects of modding (embedded images, formatting), made the process more tedious, didn't even remove irrelevant parts like kudosu or hypes, still doesn't have GDer functions or support despite being requested since day 1, and BNs got an amazing reward of 2!!!! entire months of supporter after being a BN for a year.
And then you ask what to do?
Support the fucking mapping scene instead of freezing everything since 2014 while lazer is deeeeefinitely getting released this year guys!!!!

can't really argue with any of this, we're definitely not doing anywhere near as much as we should be from the dev side of things. i've raised this internally for further discussion (citing your post directly), will see what comes of it

i can argue with a lot of it. pretty entitled viewpoint on how the game should be operating. seems to be missing the part where the whole game's development, including decisions of what gets priority, is all in the open. if something isn't getting priority, it's because there is no push for it.

how about pushing forward your proposals of what *you* think should be done to fix things, word for word, rather than expecting some magic devs to fix something which if intrinsically hard to balance, implement, and provide a pleasing experience for everyone involved?

the fact you are splitting "the community" and "the team" in your head is an issue in my opinion. you may forget that a lot of what is done on osu-web is also planned by the same community, albeit a portion that is putting in that extra effort to make things happen on the main site or client.

your perspective is entitled and a touch saddening, and i'm thoroughly confused with how ephemeral can agree with this. i'm not even taking offence from it; it's just unfortunate that with all the work i've been doing to get every system transparent (and spending hundreds of hours replying to *every last* issue thread on github across 10+ projects, regularly reprioritising / reassessing feature requests as the community bumps them).

if you are looking for specific improvements, PLEASE go on osu-web, search for the thing you think needs attention and let us know with a reply. If it's not already there, open a new issue. you can meme that this is what i say in response to anything these days, but i can't stress enough that this is how meaningful action can be taken.

btw this is a terrible place for that conversation, let's stay on whatever the topic is

oh and let me add: the people that have created the systems you mention should be proud of themselves. also i hope they had fun doing so. because at the end of the day you should not forget that: this whole thing is here to enjoy.

that should include the roles of BN, QAT and DEV.

the primary goal should be to make things fun and avoid the situation of people shitting on the system for not providing the experience they want.

adding rewards is only a band aid IMHO. keep that in mind when thinking about the next steps forward. every new system or reward we add will be temporary adrenaline which the members of the team will become complacent with over time.

keeping things changing is about the best hope we have to keep people engaged, enjoying and interested in mapping. whether that be changing the mapping scene via ranking criteria changes, adding new systems which allow different ways to enjoy mapping or modding (stuff like the mappers' guild) etc.

Cheri wrote:

Even you were acknowledge by QAT/NAT at the time with your ability to be promoted to a higher status, but yet the only thing I ever gotten was being criticised and pure hate even when the vetos/mods were justified whilst still not even considered at all fit for a higher position quite literal no reason at this point and many would agree on such.

This is incorrect. I was looked over for promotion and two other metadata helpers were made QAT in early 2018 instead. Which left me sorely bitter especially when I became the most active member later on but was stuck having to ask other people to do DQs for me. Metadata QAT were disbanded entirely in early 2018 as well. My promotion in November 2018 was not because of my contribution in that area alone but general community presence. Nor did my efforts in metadata checks ever get acknowledged besides by Kurai personally and people taking advantage of my kindness to ask me to do more checks for them. So I really mean it when I say I can relate. I had already quit doing metadata checks for months by that point.

And the only time QAH itself was relevant to QAT promotions were 2018 and entirely done away with when we changed to NAT, because QAH no longer showed proficiency in primary NAT functions..

That NAT socially have more authority is a hold over from QAT days that I'm not sure how to fix. We always remind people of the fact that in modding discussions we are equal to BN, but it never really seem to set in for community point of view. And I know you mentioned before you feel like you could do more with that authority, and while that's probably true it doesn't make NAT the right fit for you either.

While I get that having a seperate QA group could help with that issue in particular, there's still a lot of other problems we would need to adjust for first. Such as how it'd make a group of people solely for putting other's maps down, and being subject to public contempt even moreso than BNs who do the work are already. This would be a demotivator that I'm not sure would balance out any pluses.

Cheri wrote:

Having things such as meme evaluations stating something similar along the lines “oh wow you did so much QA” basically in the amidst of all of this just felt belittling, and like the work I'm doing wasn't worth anything to you.

I am sincerely sorry that our evaluation messages, which are made to be positive and make people smile about, were done in a way which offended you. I hope I phrased that grammatically correct, I really mean it. It was never the case that the effort was not worth anything to us.

That said, there was a statement earlier in the thread along the lines of how badges or titles would not be worthwhile to consider as a base form for motivators because it isn't appreciated by current QAH, but when that is a sample size of 1 that's really not an accurate view to take. Especially when things other BNs do appreciate are actively disliked by you so we can already see that people's personal preferences don't really match up there.



_________


Apparently peppy posted another reply while i was writing this so i feel like this thread is starting to split into two ongoing conversations, oh no.

peppy:
I find it hilarious you declare me to be entitled when the only reason this game makes money for you is off the back of thousands and thousands of unrewarded and unsupported work being done by bored college students in the complete absence of useful development being done. I'm not interested in discussing specific improvements because you have no clue what any of the things I listed are and will not even consider the type of reforms I would consider useful paths forward to improving the ranked section, like licensing software made by community members, paying them to develop them full time, integrating them into lazer, and then creating actual infrastructure and rewards framework for BNs and NATs beyond the current comically insulting $4 a month in supporter that they get for free. None of these things would ever happen off an osu-web proposal. Don't even try to imply that useful progress is made and not just random one-off fluff things that do not contribute long term. It's insulting to everyone that cares about the game and stays informed about what the community wants.

peppy wrote:

how about pushing forward your proposals of what *you* think should be done to fix things, word for word, rather than expecting some magic devs to fix something which if intrinsically hard to balance, implement, and provide a pleasing experience for everyone involved?

he has, and they have. i've been translating as many reasonable things into bite-sized issues that i think are feasible for short-term implementation (with my limited knowledge of what is required of osu-web scope), something that you can see in action here across the 40~ ones i've made so far. naxess has also pushed a bunch - the running trend with these is that they're proposals which don't culminate in much discussion dev side, then get forgotten.

the reason why they don't get more fleshed out is that people don't fully know where the boundaries lie (github also complicates this for the layperson), how long-term a particular solution will be (as in, how long before it is replaced with something else?) and what the endgame is for the system. there's no real roadmap for any of this, no cohesive idea of where we really should be going.

some of them have been addressed, but the meatier ones have not and this is to be expected to some degree since they involve a fair bit more than just surface level changes, especially given the history of the modding/mapping scene and said changes (aka: bouncing around from one temporary solution to another over the course of a decade, with BAT/MAT/QAT/NAT).

peppy wrote:

your perspective is entitled and a touch saddening, and i'm thoroughly confused with how ephemeral can agree with this. i'm not even taking offence from it; it's just unfortunate that with all the work i've been doing to get every system transparent (and spending hundreds of hours replying to *every last* issue thread on github across 10+ projects, regularly reprioritising / reassessing feature requests as the community bumps them).

nobody is faulting you personally here for "not doing enough" despite what nao's post might sound like. or maybe he is, idk, but he doesn't see the full scope of things. it's just that there's been no clear, concrete direction to push for specific and actionable changes that would do something in the scene for a while now - perhaps i'm to blame for that.

peppy wrote:

if you are looking for specific improvements, PLEASE go on osu-web, search for the thing you think needs attention and let us know with a reply. If it's not already there, open a new issue. you can meme that this is what i say in response to anything these days, but i can't stress enough that this is how meaningful action can be taken.

if you take anything away from this thread, the stuff outlined in this issue are at the very least a basic stepping stone for getting the reward/incentive game BN-side back to where it should be. i've seen BNs regularly getting frustrated that they've run out of supporter across a variety of channels which is an absolutely horrifying thing to witness when these people are driving like 95% of the content inflow currently. if nothing else, we should at least give them the perma-supporter (while titled) that the old BAT had, since that's basically what they are now.

Noffy wrote:

Kudosu wrote:

what about a mug?

half the reason i had that idea was because getting bns at least supporter is a thing that for some reason hasn't happened and can't seem to happen despite osu!direct enabling them to do their work more easily among other things

will look into potentially making this happen, but let me touch on a couple of reasons as to why it hasn't happened yet (from my end).

firstly, the original aim of BNs was to have the group expand out into the hundreds, and be an automatic role in some kind of modding tier system. obviously we haven't really got to that point yet, but that was the reason for not wanting give-then-take a thing.

secondly, if the supporter requirement is just for osu!direct, then that has already been resolved on lazer. it would be amazing if we could start getting modders to move over to lazer, as that is one of the early goals i would like to hit. the editor is already in a pretty stable state but things like modding have their own subset of requirements, many of which may not yet be present. if a list was made i could most definitely give priority to the remaining requirements though.

maybe it's something which will get more people involved on the development side, aka. getting features in the client itself rather than third party systems.

peppy wrote:

Noffy wrote:

[quote="Kudosu"]what about a mug?

it would be amazing if we could start getting modders to move over to lazer, as that is one of the early goals i would like to hit. the editor is already in a pretty stable state but things like modding have their own subset of requirements, many of which may not yet be present. if a list was made i could most definitely give priority to the remaining requirements though.

i would like to help give feedback for this and try modding on lazer - is there somewhere would i be able to give feedback?

There's a simple solution if you don't want to incentivize doing QAH with some form of reward, whether tangible or in the form of preparation to become a BN.

Abolish QAH and don't look back.

QAH even in a working form (where there are enough people to actually check all qualified maps) does not improve the perceived quality of the ranked section in my view. Beyond unrankable issues, mp3 quality and metadata, the kind of issues QAH finds are not relevant for 95%+ of the playerbase. This is my opinion and what I have picked up from general sentiment, feel free to disagree.

Even if QAH is done on a map and issues are fixed, the mapper always tends to apply the bare minimum fixes needed for rank. The overall quality of the mapset is barely ever improved in a noticeable way.

Controversial maps a la Super Driver or Big Black receive attention on their own, without QAH. For other maps, if there's no veto mudslinging fest pretty much any QAH changes become a footnote or forgotten.

It is already the case that perceived ranked section quality is determined almost entirely by the good will/standards of the BNs choosing to nominate maps, as even low quality maps will eventually be ranked no matter what. Embrace this, evaluate BNs harsher on map quality beyond what someone decides to DQ for, and be more strict in BN applications in regards to what the applicants standards for their own work is as well as for others.

Edit since I read the discussion above just now
@peppy
- Supporter is useful not just for osu!direct but also for beatmap upload limits
- Search filters, friend list and favourite list size as well as download speed are also nice for BNs specifically

I did some BN modding over Christmas in lazer (because it actually works pretty well on mac, shout out for that). I can try to write up a more comprehensive list but for me the main issues were
- You can't see hitsound samples on hitobjects
- copying timestamps from in-game to the site doesn't work (though that might be a mac issue?)
- can't turn off hitlighting unless I'm blind and didn't see the setting which is pretty annoying while modding

DeviousPanda wrote:

peppy wrote:

Noffy wrote:

[quote="Kudosu"]what about a mug?

it would be amazing if we could start getting modders to move over to lazer, as that is one of the early goals i would like to hit. the editor is already in a pretty stable state but things like modding have their own subset of requirements, many of which may not yet be present. if a list was made i could most definitely give priority to the remaining requirements though.

i would like to help give feedback for this and try modding on lazer - is there somewhere would i be able to give feedback?

yep! https://github.com/ppy/osu/issues is where you can do this. please make sure to read the issue template and search before posting. generally we try to make one issue per... issue, but for cases like this i'm willing to make an exception if you find it easier to list off all the point in a single issue (myself or someone else will do the splitting out).

also apologies for barging into this thread. i haven't been following closely and don't even know what QAH is, so that should give you an idea of just how much is going on in this community. hopefully the things i've mentioned will give some perspective on how i see changes happening though.

the document in the initial post is a great start and touches on points that likely need to be addressed. we will continue to support any changes that you guys see beneficial (and roll them back if they turn out not to be).

i just ask that we don't start splitting the "community" and the "core team" (as was done in the post i responded to). i can't deal with that easily, because it goes against what i *still* see osu! as being. it's not a game run by some company which is profiting off mappers. if this is your opinion then i'd kindly ask you to take a step out and reassess.

if it helps, try and understand that i think daily about disabling the supporter system, but doing so would basically destroy the game as it is. we support a huge number of artists and run some pretty insane infrastructure, not to mention pay out most of the contributors that help with contests, development, etc.

i never want this game to be about money or profit, so it makes me sick when people take on these perspectives.

i likely won't be follow this thread closely, but will continue as always to watch for any activity that comes from this on github. carry on.

@noffy

"That NAT socially have more authority is a hold over from QAT days that I'm not sure how to fix. We always remind people of the fact that in modding discussions we are equal to BN, but it never really seem to set in for community point of view. And I know you mentioned before you feel like you could do more with that authority, and while that's probably true it doesn't make NAT the right fit for you either."

Regardless of if you think she deserves NAT, or if you think QA isn't a job of NAT, there's no denying the fact that abilities NAT have (moderation perms, supporter, perceived power through having a different title) would be beneficial for someone doing a shit ton of QA.

Regardless of if "BNs are supposed to have equal authority in threads" just saying that doesn't change the fact that it isn't perceived that way by the majority of BNs and mappers.

The title alone would do wonders, and with the amount of times DQ threads get out of hand and off topic I don't see why basically the only person actively making said threads shouldn't be able to moderate them herself.

Also saying that "QA isn't a primary function of NAT so you shouldn't be NAT" is a little funny to me since there are people like Naxess in NAT who are just there for dev things which yes aid with the primary function of NAT, but aren't the primary function.

You could also make the argument that QA also directly aids the primary goal of NAT, which is evaluating existing BNs, and looking over new applicants. BN evals are, from my understanding, heavily dependent on the amount/severity of DQs a given BN gets, so assuming cheri stopped DQing maps there would be next to no DQs in general, and NAT would have a lot less to work with when it comes time for evals.

In regards to creating a QA group:

You say that one of the major issues is that a QA group would essentially be made to just put mappers down and force their opinions on them. I'd like to ask you, how is that any different from the DQs BNs make on qualified maps? You said yourself that BNs and NAT have equal power on a modding thread, so why would it be any different with a QA group? I'd like to argue it being a separate, smaller group would even make it easier to prevent unreasonable DQs considering managing the actions of around 5 people is easier than managing the actions of close to 100.

The other issue you mentioned in regards to people not liking backlash from DQs and the likes is fairly easily solved by

1. picking people who aren't as bothered by it to begin with

and

2. getting a title with moderation powers, and higher perceived authority

obviously that would require seeing if there are enough people that fit the bill, but as shown by cheri, those people definitely exist and there would be no harm in looking.

(i have more to say on the topic of issues with proposed solutions and how to deal with them, but i am trying to think more on it, so i can post something more in depth)

I mean, if the Dev team is able to consider giving supporter to BNs: what if supporter was only given to BNs who met a minimum QA activity threshold? BN are already motivated to do nominations, it's the QA that needs the motivation.

peppy wrote:

oh and let me add: the people that have created the systems you mention should be proud of themselves. also i hope they had fun doing so. because at the end of the day you should not forget that: this whole thing is here to enjoy.
that should include the roles of BN, QAT and DEV.

peppy wrote:

i never want this game to be about money or profit, so it makes me sick when people take on these perspectives.

Except you're a company in legal terms and you're profiting with unpaid third party intellectual property, be it the music or the game levels themselves. Not sure what type of illusion you put on yourself but you're a company that makes profit, properlly registered with civil responsability (Law term which idk the American English translation for).

Even from a game dev perspective, we all know that the role of the game level designers is still considered part of the actual game dev team work. Mappers are not "game modders" with third party creations added into the code of the game, they're literally making the game content itself. I won't mention and go into details about BNs, GMTs, NATs, etc, but all I can say is that the only thing you're trully safe about is the people working through github, couse that license is useful and handy for this stuff.

Also, I just want to add that this is not an attack of any sorts or me pointing fingers at you. It's just a member of this community that happens to know one or two things about law and that is also concerned that this whole thing might one day blow up in your face and leaving you wondering "What I could have done to prevent this?". Needless to say, I don't want that to happen, but again, you need to reconsider your whole attitude towards the current content creation done in this game, the people making it work and how you even perceive osu! , couse the days this software could be considered a non-profitable "fan creation" based on osu! tatakae! ouendan are long gone.

@Net0 I have no words. Take my soul and rip it to pieces. All I can hope is that your viewpoint is not shared by the majority, else maybe we should just turn off mapping.

as an aside, the BN now have refreshing supporter for the duration that they hold their title, in line with other things like NAT/GMT. this is a good step forward, and long overdue!

grabs popcorn*

also thx for the supporter

To get back on topic, I'd be interested to know people's reason why they think QAH should exist in the first place and if those goals can even be realistically achieved in any way.

In my mind removing QAH won't be much different from the current state.

Sylvarus wrote:

To get back on topic, I'd be interested to know people's reason why they think QAH should exist in the first place and if those goals can even be achieved in any way.

even though the way I wrote things in the original doc may make it sound like I think QAH should exist, I'm certainly not insisting on that. the numerous suggestions of how to possibly motivate QA was operating under the assumption that the solution would be to fix the existing system, which I don't think is the only solution - replacement altogether is also viable and should be explored. I think I mentioned this actually in saying that one of the possible solutions is to tighten standards for the performance of BNs as a way to improve overall quality.

right now I'm just waiting to see which of these options those who actually have the ability to change things are more interested in adopting as things stand.

peppy wrote:

@Net0 I have no words. Take my soul and rip it to pieces. All I can hope is that your viewpoint is not shared by the majority, else maybe we should just turn off mapping.

as the game grows so does the responsibility put on the dev team - osu stopped being 100% a passion project almost a decade or so

of course everyone involved in the scene genuinely loves the game, just look at the time and dedication every member of the bng/nat have regarding the game - or the thousands of active mappers that love making content for the game

but that doesnt take away from the fact that as a dev team you have become disconnected from the mapping scene and the overall scene of the game in general

even if it was a joke your suggestion to just turn off mapping is quite serious as this would literally kill the game

just because this isnt a small tight knit community anymore doesnt mean that its a soulless corporation, but at some point you need to stop relying 100% on the community to run the mapping scene for you when we desperately have been asking for changes and fixes for a while now - fixes that we cant implement as the community ourselves (and this is what nao is reffering to when disconnecting core team and community)

if you want an example of why this disconnect is real - just look at the bn server - the activity and discussion of the dev team is literally non-existent there (and that is the primary place where bns organise themselves)

finished my popcorn can we get back on topic?

pimp wrote:

finished my popcorn can we get back on topic?

Please do so, sorry to go on a tangent here everyone, but this was the only chance to reply to this topic.

try to stay on topic plz, i am deleting non-relevant posts to try and restore some semblance of a destination to this otherwise productive thread

plenty of places to chew us out for being tilted that aren't here

Probably has been said in this thread before but badges and supporter are great incentives for most people, see the beatmap spotlight lobby and mappers guild (which would be way less popular IMO if there was no badge incentive). Starting with perma supporter for BN is honestly a pretty good change.

More badge incentives, not sure for what exactly, would also help motivate people for tasks such as QA, higher activity, whatever else

Also maybe the elite nominator tag being given more frequently (like every year maybe?), cus it would motivate people to perform better as a BN

elite is supposed to be yearly and a big fat new batch is coming in the next few days with community contributor

Ephemeral wrote:

elite is supposed to be yearly and a big fat new batch is coming in the next few days with community contributor

Cool! didn't realize it was supposed to be yearly since I don't believe we got elite nominators last year. Thanks for the correction!

Excited to see who else gets put up as elite nominator :eyes:

wafer wrote:

Ephemeral wrote:

elite is supposed to be yearly and a big fat new batch is coming in the next few days with community contributor

Cool! didn't realize it was supposed to be yearly since I don't believe we got elite nominators last year. Thanks for the correction!

Excited to see who else gets put up as elite nominator :eyes:

I know it's been a long year but our first batch of EN were literally this time last year soo ya

Sylvarus wrote:

To get back on topic, I'd be interested to know people's reason why they think QAH should exist in the first place and if those goals can even be realistically achieved in any way.

In my mind removing QAH won't be much different from the current state.

The whole document and it's idea was under the assumption that we had to work with either literally what we got or only minor dev things

Considering the whole turn of events I am more interested in seeing if there is a different way to go about this that can be more effective and can be easier to agree upon than the previous discussion at the time, which wasn't really going anywhere.


EDIT: I am still on the fenced that having some form of QA and a mix between having more refined evaluations to increased bn quality over all is still necessary, it just a matter of which would be a more preferable approached

Because at the end even with BN quality standards changing, there will always be a need of an eye on qualified regardless unless it gets to a point where any issue thereafter can be done through a bot in which case that'll need to be discussed a bit more here

the number of qualified maps allowed per day has just increased to 10, on a vaguely related note

DeviousPanda wrote:

...but at some point you need to stop relying 100% on the community to run the mapping scene for you when we desperately have been asking for changes and fixes for a while now - fixes that we cant implement as the community ourselves (and this is what nao is reffering to when disconnecting core team and community)

if you want an example of why this disconnect is real - just look at the bn server - the activity and discussion of the dev team is literally non-existent there (and that is the primary place where bns organise themselves)

Any web changes that need to be made can be made by someone in the community who knows how to code web frontend/backend. Bounties are offered to people willing to help on the dev side. I dunno if by "fixes that we cant implement as the community ourselves" you thought someone from community can't implement changes, or if you meant that you don't think there is anybody in community with coding skills for it.

K after recollecting my thoughts and with the new situation at hands i'm going to lay out what i'm thinking about and being a bit more open-minded.

The issue currently is that we still do not have a good sustainable way of making sure we aren't too lenient on quality if we want to do away with things such as QAH

As it stands, a dq gives a lot more information and how much of an issue is relevant towards a BN more than just looking at a map that has been rank and telling them the issue after the fact.

If you are too lenient, we have the case of where we have right now, the more bns we get, the more work it is too evaluate them and any amount of feedback can be just as difficult and straining than it just looking at qualified since you have to see a massive load of maps to look over if they are doing something properly even if you just go by only 5-10 out of whatever how many they nommed.

That also not to mention that it'll be basically turn into "QA" without dqing if you think about it, which is uh arguably just as boring and more counterproductive.

And with the continuous load of bns, that too will be just as much of a hassle than it is to keep up with qualification without increasing the size of the NAT as a whole.

With no QA, evaluating bns in nature would be lenient as long as there isn’t a set standard to be place and how you guys are going to judge off of quality issues and that on it’s own needs to be discuss as the bng as a group have way too many varied opinions to have a set and stone answer for this.

And being too strict can easily cause stress on those who is generally already doing well which in general is creating something way too ideal of what it means to be a BN and still needs a set and stone standards.

A world without any QA, sounds good on paper but in reality would not work just as much as the current system relying on random dqs/bubbles and a sole person checking everything at this moment to set some form of punishment and strictness in evals and nobody can argue it ain't too lenient as of right now.

My point is regardless of what many may see, you always gonna need to have something more active to pay attention to qualified and potentially questionable things to go off regardless of what standards are put in place for evaluations.

Either that or you have the case of putting too much on NATs which let’s be honest now, their workload is going to only get bigger unless they are even more strict on things such as BN apps as the time goes on

I’m not saying QAH/QAT is needed as in like a group, but i’m just putting my thoughts on why just having no QA in general would lead to just as much of a havoc than it would good (which didn’t conclude bns getting outrage, etc.)

Tl;dr - if it won’t be just as bad as the current QAH situation now, it will later.


Now that I have that out of the way, it is clear that you still would need some form of QA management no matter how you go about this, even if it is a small group of BNs being the driving force of it all. Either that or now that we have more potential of dev support, implement something that could make this a bit more sustainable in the long run

Every single conclusion still leads off to “who will even do such?”

The core issue is still with the amount of backlash you get from even just vetoing a map regardless if justified or even just posting a suggestion on a map, that’ll always draw a line between people wanting to help out in this field, no matter the rewards

It still a great thing to push down some rewards even when considering that, as even if the rewards is not enough to keep people from doing it long, there is still the chance of those who dedicated their work to helping, should still be notice on this

Elite nominator badge and all that aside, a title + badge still wouldn’t hurt anyone regardless and I think that on it’s own can be easily implemented eventually. This isn’t really to draw attention to it but rather to just simply have it there because quite frankly, it should have been there to begin with.

The other thing is people simply burn out easily from it.

^ That is an issue that’ll never be changed unless it was made mandatory in general which isn’t really pleasant for anyone out there.

And I have no general idea how we would go about it elsewise outside of simply taking the “Q” in the title more seriously.
Regardless of whether or not the original purpose, the original purpose at large, never played that big of a part back even when this was created as unrankables was still easily missed but other quality issues were more noticed and simply put was more enjoyable to look for than being a QA bot.

Quality control is a vague thing to only go off of as we all have our standards of quality hence the veto system and mediation system.

And with those in place, people still won’t just go out of their way to just veto anything no matter how much authority they get, if it doesn’t mean they have enough support and they can easily get whatever since of power rebuked or at the very least, the mapper only lose out on a week in qualified and the map is back and going if it doesn’t get upheld.


Even for example we did make a QAH team or w/e QAT knock-off, these people will still have the consequences of being punish for doing anything stupid or too nitpicky unlike in old times and this should never be an issue to do such if they do anything wrong.

The only thing that is preventing any form of QA management being truly a good idea is just simply put, current BN’s interest and the incentive to actually maintain it which still involves needing a big enough of an gain for the BNs in question, to at least always have 1-3 people available for the specific job.

And having a minimum of 3 people maintaining it (and hey have a say in BN evals) would create more of a middle ground than too far into the left or right.

Anything else such as possibly could cause conflict such as lowering standards does not mattor too much here as other systems as cons that could be great be worse than that issue here and as long as there's a middle ground, I highly doubt that the standards can be lower than they already are in this current state.

Whether we need to create interest by having a third group or not does not matter, what matters is that we still need it and that is my point.


Sorry if that is a lot to read i'm just simply trying to make sure that we have a balance system.

I agree with Cheri that doing away with QA wouldn't be a good idea. If we do I would argue we should require every map get 3 BNs at least for rank since 2 BNs clearly aren't enough to reliably catch every issue, but I don't think that's a very good idea either.

-White's idea for the "Quasi-QAT" doesn't really seem that bad. If we remove the "stepping stone" mechanic for BNs from the original concept, there are plenty of modders who can, and are willing to mod for unrankables, assuming this is the only job this "Quasi-QAT" would have. And these modders who most likely would be a bunch of no-name modders would be pleased to receive any kind of reward or recognition for their help which BNs wouldn't normally perceive as anything incentivizing.

Why not just run with -White's idea for now, along with Cheri's adjustments? I say even throw in a supporter tag for QA members, and tie it to the minimum required activity per month or something. Monetary incentive is the best incentive.

What about long term issues? Do you really think there's a long term solution without flaws? I don't. The system could work for a year or two, really depends on the amount of new people wiling to join in. Doing something is better than doing nothing. I'm sure you will need to have this discussion every few years and decide on changes, until something sticks that is.


And concerning Nao Tomori's point about identifying unrankables vs adding value to a map. It may be that constantly looking at qualified maps might develop a quality standard in a person. And you know, people who mod maps in qualified don't only mod for unrankables, not all of them at least. There's bound to be a large portion of people interested in being a part of the QAT just for the badge or whatever benefits they will have, however that doesn't render everyone else obsolete. Even with just a small percentage, nomination team could potentially gain more members at a steady rate.

That's my side of the coin anyway.

edit: deleted last bit in fear of being retarded, but also cuz the paragraph was redundant

3 BNs would be a terrible idea. It would make ranking maps an INFINITELY harder chore, especially more niche maps that only a few bns may like.

would also reduce content load by a LOT... def better solutions to quality issues, though I honestly take a stance similar to Sylvarus where I believe QAing as a position/role is uneeded.

From what I'm seeing in this thread, it seems like the biggest issue is that BNs are missing quality issues that make a map inherently unrankable, and the systems in place for handling this problem aren't doing a good job. Does that seem about right?

In a perfect world, a program would exist which decides in finite time whether a given map has anything inherently unrankable. In this perfect world, a map would be REQUIRED to pass the check done by the program before a BN could be asked to bubble. This may not be possible mathematically (given a reasonable amount of computing space/memory - discretization means the set of rankable maps is finite and therefore, with enough space, one can write a program which checks exhaustively) with the GIVEN criteria, but certainly the criteria could be changed so that a program CAN decide in finite time whether or not a map meets it. The most obvious way of doing this is for all NECESSARY criteria for a map to be UNrankable are rigorously, objectively defined, but this isn't necessarily the only solution.

Many criteria are caught by AIMod already. Can someone with more modding experience list the criteria for unrankability that seem hard to make a program capable of recognizing? One obvious one is "diff spike in low star diffs of a spread" not being rigorously defined; another is, potentially, "incorrect metadata." Both of these can be patched (if imperfectly, but we don't reach perfection even if humans were doing QAH reliably) by having more objective versions that cover the same general cases. As an example, for the first, we could change it to: "for a map of x stars, the maximum difficulty is at most f(x)", where f(x) is some function derived by the community. Do we lose something here as compared to the original criterion? (in other words, are there maps that would satisfy this criterion that would not satisfy the original, and vice versa?) Certainly. The results of automated QA should be able to be disputed with a BN (perhaps it could cost kudosu to dispute automated QA), and a BN should still be allowed to cite the more general criteria when choosing not to bubble a map, but having this level of rigor and automation in the QA system would reduce the amount of maps needing DQ to practically zero, would it not?

Is there a reason this hasn't been proposed before?

JPK314 wrote:

Is there a reason this hasn't been proposed before?

Probably because there isn't anyone who could or would program such an AI?

wafer wrote:

3 BNs would be a terrible idea. It would make ranking maps an INFINITELY harder chore, especially more niche maps that only a few bns may like.

There's a reason I said that it was a bad idea, but the point is that 2 BNs are not enough to catch unrankables, so unless we have QA as a safety net, something else will need to be done. An AI program (imo) isn't feasible given that a lot of disqualifications are for inherently subjective things anyways (look at Super Driver for an example of that. Nothing inherently unrankable there, and I'm pretty sure there were 3+ BNs willing to nominate it).

M i X wrote:

I say even throw in a supporter tag for QA members, and tie it to the minimum required activity per month or something. Monetary incentive is the best incentive.

This is not the solution, and your statement about monetary incentive is the best incentive is just factually wrong. Money is in no way the best incentive, and the only sustainable way to motivate people is to provide intrinsic motivators rather than extrinsic. Doing that is difficult, but I definitely am strongly against offering supporter for those participating in my suggestion.

@-White Supporter only when a certain quota is reached, or maybe for top contributors. Intrinsic motivation can only get you so far.

-White wrote:

JPK314 wrote:

Is there a reason this hasn't been proposed before?

Probably because there isn't anyone who could or would program such an AI?

This is definitely not the reason. An AI in the machine learning sense is NOT what I'm talking about - the bigger part of the suggestion is applying slight modifications to the more ambiguous unrankable criteria so that it IS easy to code a program to check it automatically.

community/forums/topics/1268711?n=107 to keep relevance and waiting for a response for

I do like to say that those who keep saying it's should be abolish and whatnot, please understand that this is basically the one thing that I truly cared about more than anyone else and that I much rather be the only one doing it than getting rid of it entirely.

Quality Control was something I cared about ever since I was a bn doing qah when it was first made and I always been striving to care for it more

The whole entire thread people has been nothing but dismissive of the point and completely forgotten what this thread was even about.

Please understand the intent and that removing it is not helping me in the slightest, and if anything just remove my ability to show how much I care about my cause since other methods (like evaling i.g) is not optional and it gives me even less of an influence than I already have right now which is completely wrong.

So with that said, please let's be more productive instead of denying every single possibility to improve it.

In terms of Quality Control I'd personally probably prefer a different approach than checking every single damn mapset on a qualitative basis.

I think we'd gain a lot more mileage by digging down (Root Cause Analysis type beat) and figuring out what's really behind these common quality issues, and then looking for a broader approach? Broader being stuff like knowledge sharing, stimulating actually modding, looking into adjusting how we deliver content to different target audiences, etc; anything that can meaningfully change the environment rather than having to fight against the current. It's less controlling but ends up being a more positive effort.

Ya 100000% agree that itd best thing to do for figuring out how to solve the Quality Control issue(s) weve got going on with QA and BN nominations and all

But you stated has not in any form has address any of my issues as well as just being a utopia that can easily fall just as flat.

"In terms of Quality Control I'd personally probably prefer a different approach than checking every single damn mapset on a qualitative basis.: But completely diminishing an approach that one has been doing quite for years and not having any solid solution is completely wrong on your part.

If I wasn't ignored almost 99% of the thread, I would love to actually think of more than just one solution here but nobody has produce a single one that even relates to the topic or can be considered sustainable and that is what I do not get.

All it has been like I said dismissing one idea and not giving another one that actually can be agreed on as a whole.

doing a Root Cause Analysis flowchart or some form of workflow/process diagram in listing these issues would definitely help organize everything regardless of the feasibility of that tho

and it also looks like its more of a necessity for this as each day passes honestly, would also help in figuring out which suggestion/recommendation to try out first

yaspo wrote:

In terms of Quality Control I'd personally probably prefer a different approach than checking every single damn mapset on a qualitative basis.

I think we'd gain a lot more mileage by digging down (Root Cause Analysis type beat) and figuring out what's really behind these common quality issues, and then looking for a broader approach? Broader being stuff like knowledge sharing, stimulating actually modding, looking into adjusting how we deliver content to different target audiences, etc; anything that can meaningfully change the environment rather than having to fight against the current. It's less controlling but ends up being a more positive effort.

I feel like on paper not needing QA is a good idea, but that leads down the rabbit hole of "how do evals work if there are no DQs" and "what determines if a BN should get probation because of their noms".

While it would be nice to be able to say "well, there will still be enough DQs to judge BNs come eval time" I feel like thinking in terms of how things ideally will turn out never goes well in regards to things like this. You always have to consider the "what if", and how that can be dealt with. To combat that without a proper QA system some other form of judging BNs for evals would need to be considered that doesn't involve DQs and sev ratings.

The only thing I can think of would be judging BNs based on their ranked noms (since we would have to assume next to no maps are getting DQed), and that would essentially lead to NAT having to do QA on ranked maps in order to eval BNs which just sounds like more work from the get-go. Basing evals on ranked noms would also have other issues, such as BNs having no immediate feedback to messing up. This could very well lead to good BNs being careless because they have no indication that they missed a spread issue or a contrast issue on a low diff for instance, and then when eval time comes they will have no clue why they got probation/negative feedback. While I do think that positive reinforcement is better than negative I don't think they can exist without each other, which is why I don't think this would work.

Obviously there might be other ways to handle evals without DQs that I can't think of, but I have a feeling most, if not all, of them would be a lot of work and be very difficult to manage.

The other issue with removing QA is that if there is some magic solution that solves all the issues it will still take a lot of time and discussion, and it would be best to have something in the mean time even if removing QA is the end goal.

I feel like it would be beneficial to at least attempt something along the lines of a QA group, or at least poll for potential interest in it, considering it seems like the easiest solution to attempt and get rid of/iterate on if need be.

VINXIS wrote:

doing a Root Cause Analysis flowchart or some form of workflow/process diagram in listing these issues would definitely help organize everything regardless of the feasibility of that tho

and it also looks like its more of a necessity for this as each day passes honestly, would also help in figuring out which suggestion/recommendation to try out first

Also agree here, would really help us stop running in circles.

peppy wrote:

we will continue to support any changes that you guys see beneficial (and roll them back if they turn out not to be).

apparently we have freedom to change the ranking system as we see fit so if no meaningful changes happen it's only because the people managing the ranking system don't want to or are scared to do any meaningful change.

some concerns BNs and other users seem to have in common:

1. beatmaps ARE getting qualified with unrankable issues or obvious mistakes (because BNs are humans and make mistakes too)
2. but doing QA means hurting fellow nominators ​and "becoming the vilain".
3. because DQ's are bad.


the solution seems to be as simple as getting rid of punishment so mappers and nominators become more receptive to DQs, Nominators will be more willing to do QA, DQs will no longer be bad news and will be considered just an inconvenience at most.

edit:
the NAT should be helping BNs that are not performing well instead of punishing them, stop turning everything into paperwork and hiding behind it.

just my thoughts

first of all i see most people here talk like QA is just about finding unrankables like unsnaps, audio issues, etc

i believe that QA, as the name implies, should be about assuring the quality of the maps that shall enter rank, not check if they have basic unrankables. also nobody gets mad at a dq about 320kbps audio or an unsnap because it's just a stupid mistake that happens. and a dedicated QA group isn't needed for that matter. most issues like these already get treated in qf without an actual group dedicated to them

the moments where you can face backlash is when you'll go and dq a map for an actual quality concern (low diffs being too hard, content bloat, just a map being low quality...) and that's what the goal of a QA team/group whatever should be. and i think most will agree that there is too many maps that could've used a dq like this but ended up getting ranked with no problem

the problem is nobody wants to do that job if there isn't anything in return. if there's something in return people will do the minimum (only dq for dumb unrankables) to get the reward


tbh only thing that can be done without big dev support or feeding people with rewards to force them into doing qah is upping the standards. make BN evals more about the quality of the maps they nominate and less about their clean nom / dq ratio. punish people that push low-quality content and try to evaluate BN applicants' actual mapping standards

when the whole pp abuse drama with horrible kids and start again happened, i believe the quality of new qfed maps went up (for a short period of time). why? because BNs saw 5 others get kicked for pushing lower quality maps

there is no need for something like this to happen again but i'm sure the number of "bad" maps in qualify will decrease just by letting BNs know that yes, the quality of the maps they nominate will now matter

I propose that we allow the QAH group to add/change the RC every few months or at set points in the year. It can work like congress where members propose rules and if a majority pass it, it gets in. This system would help solve most of the quality control issues as the QAH group would be made of members who are very experienced in the issues that get pass the current RC. It would also provide a better incentive for people to try and join the QAH group and get what they want from the RC.

Currently BNs get punished lightly if a mistake is not an obvious or glaring issue from what I understand based on VINXIS' statements. However issues like snap and things on the RC are punished more heavily. If we go through with this change, we can remove BNs that aren't checking the map as they are suppose to and can hopefully limit the amount of QA needed in the future. (cause all the bad stuff would already be outlined in rc)

(^if that isn't already the case, we should punish BNs who rank maps that go against the RC harder, its a list of things to check, just go down the list and check off 1 thing at a time)

"DQS are bad" We can limit the arguments about subjective issues by using the RC. Any judgments made on subjective issues can be added to the RC by the QAH group and going forward, people can propose new rules to make all DQs objective based on the RC.

Bot wrote:

I propose that we allow the QAH group to add/change the RC every few months or at set points in the year. It can work like congress where members propose rules and if a majority pass it, it gets in. This system would help solve most of the quality control issues as the QAH group would be made of members who are very experienced in the issues that get pass the current RC. It would also provide a better incentive for people to try and join the QAH group and get what they want from the RC.

Currently BNs get punished lightly if a mistake is not an obvious or glaring issue from what I understand based on VINXIS' statements. However issues like snap and things on the RC are punished more heavily. If we go through with this change, we can remove BNs that aren't checking the map as they are suppose to and can hopefully limit the amount of QA needed in the future. (cause all the bad stuff would already be outlined in rc)

(^if that isn't already the case, we should punish BNs who rank maps that go against the RC harder, its a list of things to check, just go down the list and check off 1 thing at a time)

"DQS are bad" We can limit the arguments about subjective issues by using the RC. Any judgments made on subjective issues can be added to the RC by the QAH group and going forward, people can propose new rules to make all DQs objective based on the RC.

the issues that causes dqs other than what's already in the RC is subjective stuff that needs to be evaluated from case to case, there isn't a practical way to implement those issues in the RC

Only relying on evals/removing QA tl;dr so this stop being ignored on why this can cause a problem.

1. What are you evaluating off of? There are too many BNs with a variety of opinions to set a standard without going through convoluted loops, which would cause leniency in nature and/or be unclear from the outside come eval time, making BNs more mad because of the lack of immediate feedback that could be given by a DQ.

2. There is NOT enough NAT to support the continuous growth BNG on their own. Even increasing the standards to get in isn't really going to cut it, as time and time again that has shown it has its own problems.

3. It will just end up being like QAH without the DQ part majority of the nominations that you are evaluating.

^ How are you going to work around not checking people's nomination if you want to set a standard that everyone can follow? You still need to make sure things are in order and it just simply going to be a huge load on yourselves.

4. Giving feedback on how to do better on your quality standards is not enough and causes influx of work with the amount of BNs while effectively not stopping much of any problem because we do not live in an ideal world where everyone is willing to change, and it will just become similar to QAT, except without DQs.


Can we at least look at why QAH won't work as a Third Group/QA management branch in more detail than looking for an idealistic path that simply put won't go too far without having something far more complicated to deal with and potentially more boring than what we have right now?

Why would you want to add more work for yourselves as a small team, when you can split it up into 2 parts instead?

Why can't there be both, having more feedback on evaluations whilst also having more immediate action to prevent questionable stuff from getting rank in the meanwhile?

Why can't we test this out as now that dev support is more easier to get before concluding that this won't work?

I ask this because the arguments I had against my idea has been flat and manpower truly is not gonna be an issue if we just do a poll on it and go from there

Since right now I feel like it is better to at least attempt to test something that can make this easier for us all, than to argue in circles why something won't work here at all as I truly wish I gotten a better argument about it than what I have gotten thus far, or i'll just continue to push this idea here since I do in fact want something more immediate.

Either that or wait for solutions that can be more realistic in practice.

Kudosu wrote:

Bot wrote:

I propose that we allow the QAH group to add/change the RC every few months or at set points in the year. It can work like congress where members propose rules and if a majority pass it, it gets in. This system would help solve most of the quality control issues as the QAH group would be made of members who are very experienced in the issues that get pass the current RC. It would also provide a better incentive for people to try and join the QAH group and get what they want from the RC.

Currently BNs get punished lightly if a mistake is not an obvious or glaring issue from what I understand based on VINXIS' statements. However issues like snap and things on the RC are punished more heavily. If we go through with this change, we can remove BNs that aren't checking the map as they are suppose to and can hopefully limit the amount of QA needed in the future. (cause all the bad stuff would already be outlined in rc)

(^if that isn't already the case, we should punish BNs who rank maps that go against the RC harder, its a list of things to check, just go down the list and check off 1 thing at a time)

"DQS are bad" We can limit the arguments about subjective issues by using the RC. Any judgments made on subjective issues can be added to the RC by the QAH group and going forward, people can propose new rules to make all DQs objective based on the RC.

the issues that causes dqs other than what's already in the RC is subjective stuff that needs to be evaluated from case to case, there isn't a practical way to implement those issues in the RC

We can just do polls between BNs/Nat/whoever within a certain timeframe of a subjective issue, and go with majority rules on subjective topics, then we can add that decision on rc

QAH cant work without incentive. No one wants to be be the dude who DQs maps, its extra work and you dont get any benefit for it. Who is going to volunteer to do extra work for no benefit? (besides making the rank section better ig)

obviously we cant award monetary rewards so im suggesting we give the qah the power to change the rc (again, helps promote better quality maps and people now know what to look for better)

JPK314 wrote:

-White wrote:

JPK314 wrote:

Is there a reason this hasn't been proposed before?

Probably because there isn't anyone who could or would program such an AI?

This is definitely not the reason. An AI in the machine learning sense is NOT what I'm talking about - the bigger part of the suggestion is applying slight modifications to the more ambiguous unrankable criteria so that it IS easy to code a program to check it automatically.

"Unrankables" doesn't always refer to stuff in ranking criteria. It can also refer to subjective stuff like a slider not facing the correct direction to be create a more symmetrical pattern, a map being too bland, wrong hitsound usage based on song, or whatever other thing it might be. You basically need something that guesses what imperfections in abstract material might be and suggests how to make it better. That's a no easy feat.

Bot wrote:

the power to change the rc

Everyone has that power already, making a second, completely separate way to change the RC is just counterproductive. Discussing potential changes on the forums is infinitely better than any sort of system that's limited to a privileged group.

the RC proposal process is in a good state as-is, don't think that needs to be touched at all

abraker wrote:

JPK314 wrote:

-White wrote:

JPK314 wrote:

Is there a reason this hasn't been proposed before?

Probably because there isn't anyone who could or would program such an AI?

This is definitely not the reason. An AI in the machine learning sense is NOT what I'm talking about - the bigger part of the suggestion is applying slight modifications to the more ambiguous unrankable criteria so that it IS easy to code a program to check it automatically.

"Unrankables" doesn't always refer to stuff in ranking criteria. It can also refer to subjective stuff like a slider not facing the correct direction to be create a more symmetrical pattern, a map being too bland, wrong hitsound usage based on song, or whatever other thing it might be. You basically need something that guesses what imperfections in abstract material might be and suggests how to make it better. That's a no easy feat.

Isn't that what BNs are for, though?? I'm not saying we remove BNs, just that we remove the need for DQs by increasing the standard that reaches BNs in the first place to beyond anything DQable. Are you really saying that subjective issues that not all BNs agree upon (demonstrated by the fact that at least two BNs approve) are worth DQing a mapset over?

yes

Yes absolutely. There are subjective DQs all the time. A major reason the BNG (as I understand it) is as big as it is is to allow for lots of perspectives to be represented. Some BNs think that anything that meets the RC should get ranked, and others have subjective quality standards they think that ranked maps should meet, for one example.

So.

Over this weekend I'm gonna read through this thread and make a new post or something summarizing the major ideas + pros / cons + issues people are seeing. We can use that to help with getting back on topic and a bit more focused.

I wonder if a format similar to how let's rework qat stuff! handled multiple proposals would work? A major issue I'm seeing here is people are discussing multiple different proposals at once which makes it harder to follow and respond to in depth.

Cheri, what do you think?

Considering that the thread only has this

community/forums/topics/1268711?n=14 <-- referring an QA extension/Adding a branch
community/forums/topics/1268711?n=15 ^ Me at the same time basically stated a similar thing in detail in a document

CONS ^ was similar to QAT and tried it before didn't work/don't want to repeat anything and lack of interest still

community/forums/topics/1268711?n=59 my adjustments on -White's post
Basically about making a third group w/no BN benefits for non-bns to do QAH to add incentive instead of relying on BNs who don't want to do it.
community/forums/topics/1268711?n=46 ^ the original

CONS ^ No point in doing this after being BN, Having to trust non-bns is risky even with considering they can't dq, not has reliable as an BN, rewards can be considered iffy,etc.

community/forums/topics/1268711?n=85 <--- abolishing the system

community/forums/topics/1268711?n=126 ^ my tl;dr version why just abolishing it isn't the best idea

community/forums/topics/1268711?n=107 ^ the full version + just be me
explaining the same thing in my document a bit better

community/forums/topics/1268711?n=123 <-- refers to having QA as a group making RC proposal and such
community/forums/topics/1268711?n=130 ^ basically rejecting it as RC is fine as is


Outside of that there hasn't been much solutions as far as I see (Vinxis first post don't count as store history is already being added) and anything else that I can add here is redundant (I summarize the cons on the first 2 but it was just skimmed)

tl;dr there is basically only 3 solutions on this thread that is relevant to the topic

Well, that's a start, thank you. I'll still be looking back through everything though in case anything was missed.

we don't need QAH, at least not as a 'group'. anything that goes beyond checking for unrankables should not be considered QAH work. ensuring quality under subjective standards was a policy that was removed long ago by the staff, you can even go back to 2016 threads and you would find that the old QAT didn't actively check for issues. instead they reacted to reports from the community. this is a decision that was made by the higher ups. they effectively removed the authority to decide what fits in the ranking section from the QAT. it was decided that subjective quality control should be done by the community instead of a dedicated group of 'more capable' modders. whether this decision was a good one or not is something up for debate but that's just how it is.

now having considered that, it makes no sense to have a group of people checking for these type of issues considering that the community was already given the tools to do quality control. things like BNs being able to disqualify maps or reports notifying BNs were made for this. the people that form the QAH are no different to a regular BN so if we were considering QAH work as subjectively agreeing with the maps they mark as fine it would be no different to a 3rd nomination.

i do believe that the QAH is *somewhat* useful in its current state though. having modders checking for issues on qualified maps certainly reduces the amount of maps that are inherently unrankable hiting ranked though if we're getting an excesive amount of maps with unrankable issues into qualified then it's a problem of BNs not being proficient enough, BNs getting too confident and nominating maps without checking them or them not having the right tools to avoid these issues. we could for example have something like mapset verifier run on the osu web when pressing the nominate button so that it minimizes obvious mistakes, have some sort of version control etc.

i think that in the end, if we're considering checking for unrankables only then it's a problem that should be tackled from a different perspective. if you really want to have a group checking for more than unrankable issues then it's something that needs to be supported by the staff. enforcing quality in a subjective manner is not easy especially considering how easily a mod can be rejected these days. mappers and modders are aware of this and they know that continously denying a mod will eventually lead to the modder giving up on the issue and move on since they're just wasting time. if a group of people were to do a job like this they would need to have some sort of authority to deny maps from getting ranked (this has already been done in the past and deemed as not satisfactory).

i think the process of doing QA is similar to evaluating BN candidates (download the map, identify mistakes, report them).
the biggest difference is that you spend more time with the evaluation maps but you check fewer maps, while doing QA you check more maps and spend less time with them.

in reality you will not find issues on many maps during QA checks. it's still a job that needs to be done because the maps with issues do reach qualified.
it's not only a a few members of BN/NAT that are missing these issues, there's definitely those who manage to be more consistent at performing clean nominations but there are just so many things that can be overlooked... everyone miss something unrankable or obvious eventually.

like i mentioned before, i think doing evals is similar to doing QA but why are BNs doing evals and not QA? because there's an incentive to do evals and this incentive doesn't exist for QA(+ everyone hates DQs)...

for that reason i believe the ideas from the proposal could work indeed as long as it focus on what QA means currently (checking for unrankables and obvious mistakes)
instead of just being a new generation of QAT (performing all DQs, being the only people responsable for vetoes etc)
i think that every member of BN NAT should keep their ability to DQ maps and perform vetoes instead of moving this responsability to the new team/nat.
while this new team/nat would focus on what QA means currently, they would still be able to perform vetoes but they would be performing them individually and not as a team.

I edited original post with the only proposals (that was relevant to the topic) on this thread.
There is no reason for a new thread as there was barely an actual discussion to begin with + this is now pinned.
-------------------------------------------
Now on topic

I do like to see a little adjustment on the BN site on how QA checks are being done

Right now on the site
https://i.imgur.com/e7y8sFz.png


^
This is pretty lacking as it is currently and I think some additional tools would be useful



---


Generally a lot of above is nothing more than QoL changes as well as it can help make QA a bit more functional at least on the BN site and more pleasing.

Also this goes to show how much I do think that QA should be considered as a NAT branch, being that this is rather more direct vs indirect feedback from evaluations and can easily split the work if done correctly. Either that or the third group.

---

I think the biggest thing in this thread is that my intent isn't really perceived the way I was trying to go about this and I think the biggest thing is that people are looking at this is if I am asking for QAT back which is rather not fully true in the sense that people is questioning it for.

So with that, I will actually explain my case so we stop the argument "it just QAT and because we did it before, we shouldn't do it again."

Rather than enforcing quality standards upon maps and such, what I'm looking for is simply "assured" the BN on their nominations, and simply give more active feedback in a more frequent matter while checking for unrankables/might as well be unrankable (spreads i.g).

The thing is whilst stuff like probation exist, it isn't stable or what I would considered a consistent way of handling quality over all likewise with giving feedback during evaluation period since it's 3 months after the fact.

One of the things that usually do more outlasting impact is dqs that changes the perception of what is okay or not and apply punishment during these times. But these outcomes such as the incident that got a lot of BNs kick, is that it is rather harsh which is not really a bad thing, but the harshness only implements a scared temporarily as most know things get slowly back to normal over time if that harshness/strictness is not continuously pushed.

That is why along with making the QA page more functional with stuff that should of been there in the first place, I believe it could be more useful and actually brings more of a positive rather than a straight negative.

Having an active QA giving a "review" on a map and their nominators mod instead of just straight up looking for issues can easily change the perception and actually creates a more useful feature to looking at the qualified section.


Because right now as MV and all that is getting more and more useful and possibly we will have things such as a bot to moderate unrankable issues, it can be easily seen why QA is seen rather a boring task even though it is basically similar to evaluations especially with the inherited negative affect it gives from the start since the focus is on "DQs" and such.

Even changing the idea of making it more subjective focus with vetos, can create problems with people as nobody likes to have a DQ even if it is useful to the map, due to a lot of factors with how the system is naturally with the main one being the backlash.

So by the idea of a review, I think simply having the comment part become basically a way to give feedback to the nominator whether it be positive or negative would be highly useful.

Obviously the main focus of QA is still for unrankables, but giving way for QA to tell the nominators they are doing a good job and how well their mods is on the map would give more positive assurance to the nominators and make it less of a nervous wreck when their nominations gets DQ'd at times.

^ Likewise with focusing on criticizing how unpolished the mapset is and how the nominator could be potentially slacking without necessarily dqing the nomination, sets a different tone than a straight up veto/putting the suggestions in mapset itself at the time.

Also this straight up makes it less boring in a sense of just focusing on unrankables, but also doesn't scared people from having a say since it isn't as dq focus as it is currently.

Considering we do not want this to be completely subjective focus, having it more as a way to self-reflect is better in this case and to also keep them on the right track so they could work on their nominations in the future and having DQs that is subjective taken more seriously when it is actually serious or slowly get to that when the negative backlash is lessen over time as nominators/mappers would see the intent is less about pushing/enforcing standards.

Basically leave subjective suggestions to the BNs in hand while QA still follow the job of just making sure things are in order and to help guide the BNs from trouble in the future.

By doing all of above, I think this way of giving active feedback on nominations is a natural positive incentive for nominators alike as it won't apply pressure to them, but instead just keep them from falling short.

The issue with this though is that, It is more work on the QA part to do so.

If QA was an active branch in NAT or Group of it's own, then they can easily focus on this without nominations/evaluations on their own/etc. As well as it just simply gives a better tone for a person who has a title and is meant for this vs a BN as the intent can easily be misunderstood.

Keep in mind that just like evaluation, QA doesn't have to give a review on every single map in qualified as that is insane, but they should still check a healthy amount of maps regardless for unrankables/other issues still.

With a group of 3-4 people and having at least 2 reviews required per member a week and checking at least 4 maps for other issues, would be a nice balance currently.

Over all, I do think this would actually create a positive incentive naturally on it's own and remove some of the negative features of doing QA such as it being bland.

While it is true it will change the intent of QAH but that literally doesn't matter because as long as it is an boring incentive even if it is useful, people will continuously ignored it especially being it also have a negative side to it with backlash.

And changing it where it gives less work for the NAT (as if QA is considered basically something to work together with them), would also create less negative vibes as a whole if it is more improvement focus vs punishment focus and only push punishment when the nominators simply still rejects feeback/still careless.

I hope with all of that I've said, my intent is more clear to anyone who read this thread and with that I am done updating this post (Except for fixing grammar issues.)

Has any further discussion happened in regards to any changes being made moving forward? Or have any conclusions been made?

The way things are now seem very unsustainable, unfair and unfun with a lot of things in the current system causing a lot of stress for nominators. I feel that some change to fix the issues outlined in cheri's doc and other issues in the bn system (ask them not me) would be very healthy for the whole mapping community which would then also benefit players. At the very least, some sort of clear verdict or a further response from NAT would help to elaborate the official stance on these topics and the goals the current systems are trying to achieve.

Lots of good ideas have been thrown around and Cheri did a great job summarizing the relevant proposals. So are we going to see any direct action?

Its really disappointing to see so little personal initiative by most of the NAT members considering Cheri has been carrying QA on her back for so long now. Regardless of the outcome, there needs to be more effort to discuss these issues effectively, and come to some sort of conclusion.

Just to add a little on this aswell:
I think the idea of QAH in general is good. Having people check for unrankables in the qualified section before maps hit ranked should definitely still be done, but differently. I can agree with pretty much everything that cheri has stated towards the QAH and with the community just growing and more mappers ranking maps, it's definitely needed to have some form of quality assurance, since that is what qualified is for in the first place and having a dq happen (which will probably happen more often once newer mappers rank their maps) is much better than having to unrank a map.
With the suggestions that cheri made here I think it's important to adress them, because they have a really valid point that shouldn't be overlooked.

it takes time to formulate results for these kinds of proposals (especially in this sort of volume). the issue is being discussed though and i'm sure there will be movement on the matter eventually, though i doubt it'll be a 100% pull from any of the proposals floated in here, since that's generally not how things work.

that being said, this thread is a super mess. would be nice to get some bullet point summaries of what the issues are and any proposed solutions since i imagine i'm not alone with my head swimming at the prospect of having to pick through this 140+ response thread and several gdocs to see where stuff is at

Cheri wrote:

Considering that the thread only has this

community/forums/topics/1268711?n=14 <-- referring an QA extension/Adding a branch
community/forums/topics/1268711?n=15 ^ Me at the same time basically stated a similar thing in detail in a document

CONS ^ was similar to QAT and tried it before didn't work/don't want to repeat anything and lack of interest still

community/forums/topics/1268711?n=59 my adjustments on -White's post
Basically about making a third group w/no BN benefits for non-bns to do QAH to add incentive instead of relying on BNs who don't want to do it.
community/forums/topics/1268711?n=46 ^ the original

CONS ^ No point in doing this after being BN, Having to trust non-bns is risky even with considering they can't dq, not has reliable as an BN, rewards can be considered iffy,etc.

community/forums/topics/1268711?n=85 <--- abolishing the system

community/forums/topics/1268711?n=126 ^ my tl;dr version why just abolishing it isn't the best idea

community/forums/topics/1268711?n=107 ^ the full version + just be me
explaining the same thing in my document a bit better

community/forums/topics/1268711?n=123 <-- refers to having QA as a group making RC proposal and such
community/forums/topics/1268711?n=130 ^ basically rejecting it as RC is fine as is


Outside of that there hasn't been much solutions as far as I see (Vinxis first post don't count as store history is already being added) and anything else that I can add here is redundant (I summarize the cons on the first 2 but it was just skimmed)

tl;dr there is basically only 3 solutions on this thread that is relevant to the topic

@Ephemeral im p sure that's what this was for, lists proposed solutions and their pros/cons

Ephemeral wrote:

it takes time to formulate results for these kinds of proposals (especially in this sort of volume). the issue is being discussed though and i'm sure there will be movement on the matter eventually, though i doubt it'll be a 100% pull from any of the proposals floated in here, since that's generally not how things work.

that being said, this thread is a super mess. would be nice to get some bullet point summaries of what the issues are and any proposed solutions since i imagine i'm not alone with my head swimming at the prospect of having to pick through this 140+ response thread and several gdocs to see where stuff is at

I made a post if u scroll above as a summary of it and put it on the original post 4 days ago. I didn't add anything that was just redundant or didn't really add much more than what those currently stated.



While I get that they wouldn't straight up take the proposals here 100% and while I understand it takes time,
BUT considering that there is very little of actual discussion around this thread in general along with the fact there is only 4 proposals related to this thread thus far is highly ridiculous to not be talking more in this thread to discuss for further things publicly.

And it just not 4 proposals, it also that 2/3 of them is very likely not going to happen as it is too faulty from the get go (one is literally about abolishing the system and the other was hard denied so uh .-.)

A response is kind of needed to see what they would go about things or if they would in general at the very least soon being that for all I know, they can go an entirely different route due to it being basically only 1 proposal without too many cons to go off of to do a change...

I agree with cheri here, there wasn't much actual discussion or proposals made in this thread (as evidenced by the summary she made), so there's not much to work with. There were really only 3 actual proposals made, and one of them is flat out removing the system (so there isn't anything to pull from there). While I do think under normal circumstances pulling from other people's proposals/ideas is the best way to go about things, like you said, I really don't think that's an option as of right now, which leaves is with two options:

- More indepth discussion on really the only remotely viable proposal

or

- Some other viable proposal to be made within a short timefarme (which isn't very likely)

Frankly, the latter should have been done much, much earlier into this thread, but people were too caught up whining about the origional "proposal" which really wasn't even a proposal at that point. If I have to be blunt here, basically all of the "discussion" done by NAT in this thread was extremely unhelpful and quite unprofessional. The least they could do at this point is discuss the, essentially, singular proposal we have.

Hi, just a short update from my end of things since I get the silence can be a little deafening.

A few of us have been bombarded with questions the past couple of weeks about whether we are going to do anything. Please be rest assured that we haven't forgotten about the concerns brought up in this thread. While we're still unsure of a solid plan to go forward on, most, if not all, of the NAT agree that some changes are necessary.

As it is right now, we're unlikely to take any of the proposals outlined in this thread as is, however that doesn't mean we're going to dismiss them completely. There's currently some plans behind the scenes to put together alternative solutions, and while I can't give an ETA, we aim to get everyone's input on it as soon as it is reasonably possible to do so.

All I'm asking is to just sit tight for a little bit longer.

-Mo- wrote:

Hi, just a short update from my end of things since I get the silence can be a little deafening.

A few of us have been bombarded with questions the past couple of weeks about whether we are going to do anything. Please be rest assured that we haven't forgotten about the concerns brought up in this thread. While we're still unsure of a solid plan to go forward on, most, if not all, of the NAT agree that some changes are necessary.

As it is right now, we're unlikely to take any of the proposals outlined in this thread as is, however that doesn't mean we're going to dismiss them completely. There's currently some plans behind the scenes to put together alternative solutions, and while I can't give an ETA, we aim to get everyone's input on it as soon as it is reasonably possible to do so.

All I'm asking is to just sit tight for a little bit longer.

While I understand that things takes time and that you all need to gather your own thoughts to make a post to get everyone's input and whatnot, the thought of literally ignoring my own statements around this should be talk more and discuss more in general is unreasonable to me especially considering as far as I am concern, I am not even sure if these "alternative" solutions would actually even solved the case and I hate the idea of waiting a while to give input around that.

and by discussion more around, I mean something reasonable instead of how u guys having it internally should be put in place like using the #modding server so you have input as you are making alternative solutions rather than waiting until the last minute or something close resemblance.

We literally had these problems for months so I am expecting more than just a "wait more" response here.

-Mo- wrote:

All I'm asking is to just sit tight for a little bit longer.

Not to be impatient or anything but I feel this is a pretty major issue and there's been some decent suggestions made in the thread that haven't been addressed properly at all. We all agree that changes are necessary, so some kind of update might be nice after 3 months.

The lack of a sense of urgency is disappointing, at least to me. Even if the suggestions proposed in the thread aren't perfect, they're probably better than the current implementation (or at least worth trying) until a better solution can be implemented.

-White wrote:

-Mo- wrote:

All I'm asking is to just sit tight for a little bit longer.

Not to be impatient or anything but I feel this is a pretty major issue and there's been some decent suggestions made in the thread that haven't been addressed properly at all. We all agree that changes are necessary, so some kind of update might be nice after 3 months.

The lack of a sense of urgency is disappointing, at least to me. Even if the suggestions proposed in the thread aren't perfect, they're probably better than the current implementation (or at least worth trying) until a better solution can be implemented.

6 months and still sitting tight eh ?