Before reading this post, I wanna say this has nothing to do with me getting probationed a few days ago, in fact this is something I already had in mind proposing before I even got BN.

How do evaluations work in the current system?

An evaluation is made by three different members from the NAT (usually) by going through the voting -> discussion -> consensus phase in order to evaluate a member from the beatmap nominators. Card evaluations are automatically created for each member of the team after 3 months from their previous evaluation (it goes up to 6 months if you've had 2 successful evaluations in a row). Probationary BNs go through an evaluation every month instead.

When going through an evaluation, you will get a pass/fail consensus that will decide whether you stay in the team or not, you may also get moved to the probationary BNs or get an extended period if you are already one of them and sometimes get an inactivity/quality/behavior warning.

Besides everything I just mentioned above, NATs can generate evaluation cards anytime for any member of the team if they feel like it's necessary (this is usually only done in emergency situations like a BN breaking essential rules). For more detail on everything I recommend taking a look at the wiki.

Why evaluating Full BNs is not worth the time

Full Beatmap Nominator evaluations end up pretty much always in a pass consensus which makes it sort of useless for the NATs to spend time evaluating when they could be focusing in other more relevant stuff instead like evaluating BN applications (application responses have been extremely delayed over the past months with +30 days of waiting time when it always used to be <2 weeks.)

When taking a look at failed evaluations (mostly BNs who got moved from full to probationary) we can all see that they suffer from the same traits (a high number of disqualifications/high disqualification rate compared to the number of nominations + high severity values in most of these). This alone should be more than enough to understand who is not doing their job properly instead of having to evaluate individually each one of the 100+ members that are in the team.

Conclusion

Make NATs evaluate at their own accord by manually creating evaluation cards only for those who may seem like are not doing things the right way (usually spoted by disqualification + severity rates). Incentivize the use of reports so that NATs can be more aware of single exceptions (behavior, inactivity and quality stuff).

Would like to see a lot of variated opinions from everyone (specially you, NATs) to see what you think about the proposal ^^

idk if passing the full BN was as easy as you make it out to be the evals should not take so much time. If the eval is taking time to go through it probably means it isn't as clear cut as it seems behind the scenes (need NAT to confirm since I don't know what exactly happens)

Also i think we need more quality enforcement, not less lol. I get that it's a huge chore and hassle to do so many evals so frequently but right now it takes more effort to get BN rather than to stay BN. (Getting activity, prepping app mods etc. vs not inting ur noms and dont get dqs lol)

Right now it is too easy to just coast as a BN. Just get minimum activity and post some random shit in #absence-notice to soften ur eval if u int too much. Barring anything that could cause an instant eval (unranks etc.), it could take up to 3 months before someone gets probationed then 1 more month b4 a kick. This means that there is up 4 months for a potentially unqualified or a BN who doesnt give a shit to stay as a BN and continue pushing maps.

I like the idea of putting more focus in manual eval cards thru things like DQ rate or user reports, but I feel like having more strict standards in the regular evals are needed, not removing them entirely.

but the NATs r human too, with lives outside of the game. Honestly the problem is just the NAT to BN ratio (esp for STD) is not that ideal right now. But there isnt rly a good solution for that is there

tl;dr: what u said kinda makes sense but i feel that we need more quality enforcement, not less


Edit: one thing I want to add is that dq rates arent the only thing that matters in terms of BN performance. Regular evals allow NATs to evaluate a BNs modding quality. With QA in qualified being pretty much dead (until recently it seems) it is possible that maps go through QF to rank without any issues when in reality the modding is subpar and said map could use more thorough modding

Idk what it is like recently since i just rejoined, but a year or two ago some of my pass evals had feedback and tips in them, which i think is useful in terms of continously raising the standard in BNG

Honestly I don't think it'd be any better to make NATs evaluate on their own accord as this has a high chance of certain things going unnoticed (as there is now but regardless). This'd also *arguably* require more work from the NATs as they wouldn't just have eval cards generated every 3 (/ 6 months as some full BNs get nowadays if they've gotten 2 positive evaluations in a row) which could also burn out some of the NATs even more.

Conclusion: Keep the current system with an eval every 3 months and 6 months for those performing well as some sort of routine checkup is always good.

(edit: +1 to kotoha's quality enforcement thing)

????

full bns should always be evaluated, the only reason it's almost always the same result is because nat doesn't want to probation / kick people for nominating garbage left and right

Full Beatmap Nominator evaluations end up pretty much always in a pass consensus'

This is correct but other ppl have expressed their opinions. Almost the same as mine.
so I won't go in depth and repeat everything again.

Scheduled evaluation is *always* necessary to make everything clear, fair, tidy and in order IMO.

At times we spot things that don't show up as result of any kind of DQ rate or SEV moment. This alone makes a routine-check useful.

Doing things without a routine check-up makes the entire thing an information game rather than a transparent system. There's a plethora of issues this brings for a NAT-BN relationship that I see as based on trust.

The fact that a lot of BNs simply pass creates this trust and confidence in that the system is doing well. Without regular check-ups we could be nothing but permanently skeptical about the bits and pieces of information we do take in.

In general I don't think of this as a good idea.

I'd also like to draw attention to the following statement

Didah wrote:

application responses have been extremely delayed over the past months with +30 days of waiting time when it always used to be <2 weeks

so .. you're spotting an exception to the general rule and conclude we should tackle it by nuking an otherwise functional system? I don't know what the name for this would be but it definitely reads like a logical fallacy

Mordred wrote:

the only reason it's almost always the same result is because nat doesn't want to probation / kick people for nominating garbage left and right

this basically

there's also the factor that issues can get ranked going unnoticed in qualified (there have been plenty of these before like silenced actives, missing hitsounds, misplaced rhythms, etc) so saying that dq rates and their sev ratings should be the main factor is not it, you still need someone to recognise these even if they get ranked... doing checks semi consistently can get you a higher chance of it getting actually caught relatively in time in stead of letting it go unnoticed
^ not to mention the quality of the maps themselves Lol

imo trying to get nat to uphold a higher timing standard for themselves could work (heyy my eval was due 3 days ago)
moving a handful of competent bns to nats could work to reduce the workload across the nat though i believe the thing they're doing right now with 6 month eval periods for well performing (!) bns is already a step in a good direction

yaspo summed it up perfectly

if you're concerned about long delays then I think re-instating the BN evaluator system with some adjustments would help a ton.

Some adjustments that could be made:

1. Make it invite-only instead of it being RNG like last time.
2. Enforce stricter activity requirements on them, so it's easier to cycle out inactive/barely active evaluators.
3. If you really need to use a cycle system, you can have 3-month cycles and allow evaluators to stay for 4 consecutive cycles at max, so a good evaluator would get to eval for a year then have a 3-month break, and repeat.
   Could even include some kind of reward as an incentive, like a badge for performing well for 4 consecutive cycles.

Edit: moved the discussion for the BN evaluation system here: community/forums/topics/1580704?n=1

archiving in favor of community/forums/topics/1580704