Greetings,
Listed here are the tasks and conclusions derived from the 5th NAT meeting, "Current BN Evaluations".
You can find future and past meeting topics here.
Activity Thresholds
How Resets and Activity Affect Evaluations
Non-task Conclusions
If we've missed a task related to the meeting, or you have questions or concerns about existing tasks, feel free to post in this thread. We won’t follow it very actively, however, so responses may be infrequent.
Listed here are the tasks and conclusions derived from the 5th NAT meeting, "Current BN Evaluations".
You can find future and past meeting topics here.
Activity Thresholds
- Document the following changes to thresholds:
- Standard, taiko, and catch BNs must now do at least 3 nominations per month, up from 1
- Mania BNs must now do at least 2 nominations per month, also up from 1
- Mania being an exception due to few BNs otherwise able to keep up
- Implement automatic detection of inactive BNs for manual review at a frequency of once every 2 months
- Should average activity over the last 3 months to account for irregularities
- Document above changes to frequency
How Resets and Activity Affect Evaluations
- Trial having the evaluating NAT take poor mapping quality into account if most agree
- Reason for trialing is to see how NAT opinions differ and/or how inline this is with BNs' expectation
- Eventually implement a way for QAH to mark maps for potentially poor quality which NAT can review in evaluations (if the above trial works)
Non-task Conclusions
- The following were considered working as-is by participants and need no immediate changes:
- How issue severity is judged in evaluations by NAT
- That activity is judged from unique nominations and modding
- How greater nomination activity offsets resets to some extent
- Specifically using a 1/3 resets to unique nominations ratio as a rule-of-thumb threshold for performance
- Unranks no longer make a difference in how severe resets are considered in evaluations
- Related to vetoing poor quality maps as a way to mark them as such for evaluations; participating BNs believed that most poor quality maps entering qualified were simply poorly modded rather than inherently flawed, and that vetoing would be overkill
- We'll follow up on this in a later meeting
If we've missed a task related to the meeting, or you have questions or concerns about existing tasks, feel free to post in this thread. We won’t follow it very actively, however, so responses may be infrequent.