forum

[Proposal] Improving and clarifying the current probation system

posted
Total Posts
6
Topic Starter
squirrelpascals
Probation was created as a reevaluation for bns with a potential to improve, but is now becoming more seen as pressure to avoid iconning maps that could receive dqs or bubble pops. With changes to the QAT system right around the corner, how probation is currently handled isn’t something that should be further neglected.

The current system

Very generally, the probation system goes like this:

  1. Bns are notified that evaluations are starting
  2. Based off of modding performance evaluated by QATs under a case-by-case basis, if you have underperformed then you are placed on probation
  3. During probation, go about your bn duties with knowledge you’re being evaluated
  4. If you’re still underperforming by discretion of the qat, you fail probation. If you’ve improved you pass probation. If you weren’t active enough, usually this will get extended.

This is what probation appears as on a surface level. It’s not a secret that DQs and bubble pops largely influence your evaluation. This isn’t unreasonable, especially if you’re getting consistent pops or dqs. But there’s no telling what proportion of bubble pops or dqs count as “underperforming.” As a result, bns are pressured to perform as closely to perfect as possible, by nominating safer maps, or maps from more experienced mappers, even if it’s against their will. This is the root of the problems with the system.

It’s also not a secret that more than just dqs or bubble pops are being considered. Reviewing on a case-by-base basis makes sense, because it’s not a one size fits all situation. But it isn’t doing us any good if we don’t know what specifically QAT / management are looking at when we’re being evaluated.

Personally under probation, I felt like I only vaguely figured out what was evaluated, based off a mixture of rumors and experiences from other BN and QAT. I myself had my previous reasons for being kicked to improve upon (which still felt vague) but I couldn't see anything for newer bns to focus on. This whole ordeal made probation super super stressful for me, and lead to me trying to be perfect, which I know isn't a realistic expectation. Also, it made it a bit of a mystery as to why some of the people didn't pass. Probation doesn't need to be stressful or mysterious.

Other perspectives

Most of these problems can be fixed with more transparency. I’ve talked with 6 other people who have gone through the probation system, 5 from those who have passed. Despite this, 4 out of 6 people found the process to be vague and provide a lack of transparency, overall viewing it as a stressful experience. The other 2 perspectives both passed probation without seeing it very negatively, and also received little to no dqs or bubble pops. Given that every bn will go through probation at least once (when they join the bng), 4/6 is not a good proportion of people who see probation to be overall negative or stressful. Since new bns are welcomed into the bng under probation after all, if a bn is to struggle through probation, it doesn’t make the bng seem very welcoming.

Proposed changes to the system

  1. Write a set of “probation guidelines” that cover what the QAT / management will use to evaluate you

    We’re told generally throughout our bn careers that our activity, attitude, and our ability to successfully push maps to ranked is being considered, but we already know that. We don't really know what skills we need to focus on obtaining in order to bridge the gap between being a probationary and a full bn. Proficiency in finding unrankable issues? How active should we be? etc.
    These thing need to be written out formally and should be posted with probation warnings (next proposed change), notifications of bn evaluations, and would also be very useful to find within the bn guidelines.
  2. Warn bns that they’re in danger of being put on probation so it doesn’t come across as a surprise

    I got this idea from somewhere else, I forget where. Approaching the date of your revaluation results, there’s a lot of uncertainty as to whether or not you’re doing everything correctly, which in a way makes the revaluation feel like a doomsday. It would be a lot more helpful to have some final warning that you need to improve something before you get surprise-attacked by probation status.
  3. Give detailed information as to why each bn is being probationed

    How can you know how to pass probation if you don’t know what to improve? I mentioned earlier that probation isn’t one-size-fits-all but due to the vague process, that’s how it feels like it’s being treated right now. Bns will likely perform better in probation if they know what they did wrong beforehand, and know what they can further improve by the time they’re evaluated again. To compliment this, bns should be able to openly discuss their evaluations with QAT / management person on the subject of their evaluations. (If evaluations happen one month from the evaluation announcement, this can happen like a week or two before the evaluations happen).
  4. Consider bubble pops less than dqs in evaluations

    By weighting bubble pops as evenly as dqs, I don’t feel like we’re thinking about the reason why we have a 2 nominator system in place. In a two nominator system, the two different bns’ perspectives basically ensure that each map is reviewed as thoroughly as possible; if bn #1 doesn’t find an issue with a set, by the time it gets qualified bn #2 is more than likely to have found that issue, because two bns are likely to have different mindsets and processes when modding a map. No map will be perfect after the first bn mod, and dqs are generally seen as a "bigger deal" in most peoples' eyes anyways.
    Additionally, nominations are reset pretty inconsistently between a number of different issues, especially when it comes to issues such as missnaps, tags, or spreads, so it would add a tiny bit more fairness to the process.
  5. Don’t consider dqs or pops for complex timing in evaluations

    Or at least consider them less. Complex timing is not an easy to come by skill that shouldn’t be realistically expected for every bn. Usually when you’re told to ask someone about timing on a map, you hear the same 5 or so names thrown around. I’ve even heard them described as “scary” to nominate because nowadays timing dqs are almost inevitable once your map gets qualified.
    To be clear, simple things like offset and blatantly wrong timing should be taken into consideration. Timing often gets dqed for pretty trivial, small improvements however which makes these maps less attractive to bns.
  6. Add the ability to request extensions ahead of time

    This is kind of an afterthought to what I talked about in most of the post, but I’ve seen a lot of people become worried about their bn status if their on probation while a big event is happening in their life, or recently, if they had to judge for MWC / some other big event (mostly through #absence_notices in the bn server). Real life should always take priority over circle clicking. It would be a lot more relieving if you were able to tell a QAT / management person that low activity was to be expected, and request be evaluated next month instead.
    Of course if this is a thing, I feel like more limitations would be needed on that prospective bn.


Google doc link, where I wrote most of this
Discussion about potential changes would be awesome.

*Changed "QAT" -> "QAT / management" in some places to refer to newer system
Naxess
Think a lot of this will come after the rework since who knows what'll break or no longer function properly by the time that rolls around with all the changes that are coming. There's the possibility that this whole thing will be replaced by another bn score system as well, so digging too deep into it currently, with the little transparency we have into the coming changes and how/when they'll be accomplished, is probably not a great idea. However, we can still discuss it until then (which should be soon, from the looks of things).

Write a set of “probation guidelines” that cover what the QAT will use to evaluate you
A lot of it will likely remain vague since much of it is case-by-case judgement based on your activity in modding/nominating/resetting stuff and getting reset, and that all depends on what you're reset for, if it's something you're reset often for, if it's something you were warned for, if it's something most people would miss, etc. So this may be difficult to define, but probably doable nonetheless. Pretty sure examples will need to be a big part of this; just writing them like guidelines isn't going to work well.

Warn bns that they’re in danger of being put on probation so it doesn’t come across as a surprise
For full bns (since new will already be on probation) that sounds great, but there's a huge workload in trying to gauge that in frequent enough periods to be able to effectively warn people for it. Right now, the probation is already in itself acting as a warning, and if it's something you haven't been extended for recently (~6 months, depending on case), you'll very likely be extended again unless you somehow mess up horribly. So it's basically a 2-strike system but with fading and category-dependent strikes.

Give detailed information as to why each bn is being probationed
Deliberations sent out to probation bns is used as reference during their following evaluation, with repeated issues of specific details being weighed more heavily, so if it isn't specific, chances are you will pass more easily than otherwise by just being more careful in general. That's not to excuse things not being specific, because we do try to provide specific details for what each bn should focus more on in every deliberation, but it does make things more fair by having a safety net like that. For what it's worth BNs can already openly discuss their evaluations with the QAT who judged them if they ask for it (or at least that's how I understand it).

Consider bubble pops less than dqs in evaluations
The reason we don't care about whether it's a bubble pop or dq is because either can be more or less relevant depending on context. If a bn gets popped for missing that the map is actually 30+ offset (and not variable bpm), that's really bad, and another bn missing that too doesn't make it any better or worse, just that there are now 2 bns that missed the issue. Disqualifications being seen as a "bigger deal" isn't relevant to this. If the issue isn't something we would dq for, it's very likely not going to be considered.

Don’t consider dqs or pops for complex timing in evaluations
Not considering this at all means people would just stop caring, or care less, about it. Complex timing maps inherently take more skill in timing to judge properly, so it would be preferable if bns experienced in timing would look at those more than those not as experienced in it from a pure preventing resets/unranks perspective. I do agree that it should be taken into account more, though, as in being weighed less. Some QAT already do, but including it in the data we gather would probably make the situation better.

Add the ability to request extensions ahead of time

If you haven't been extended for activity before you could probably take a hit on that to extend the probation for a month. In the case of needing to judge important contests or whatnot we can delay probation periods further, and is something we did recently, but you'll still land on probation for activity afterwards (and during the period you're delayed, you can't pass probation either, since you're not evaluated). So the whole automatically being put on probation for activity when doing this is the limitation. Also note that you can probation for timing -> extended for timing -> extended for activity -> etc, as long as each extension is for a different thing and the previous thing improved enough or couldn't be judged.

I realize half of this should be in the clarification post when/if it's made, but it's kinda hard to discuss the system without talking about how it works
anna apple

squirrelpascals wrote:

  1. Write a set of “probation guidelines” that cover what the QAT / management will use to evaluate you

    We’re told generally throughout our bn careers that our activity, attitude, and our ability to successfully push maps to ranked is being considered, but we already know that. We don't really know what skills we need to focus on obtaining in order to bridge the gap between being a probationary and a full bn. Proficiency in finding unrankable issues? How active should we be? etc.
    These thing need to be written out formally and should be posted with probation warnings (next proposed change), notifications of bn evaluations, and would also be very useful to find within the bn guidelines.

    To some degree this is feasible, but at the same time not so much. I think Naxess's main issue comes from "well if we miss something: we are liable for promoting a guideline(s) list and such a problem/issue not being reference-able in such list. At the same time I think there can be more direction in personal BNs, especially in the system proposed by UC and changed up a bit by QAT/Mao where this is more feasible considering workload will be pushed more onto BN management than other tasks. (But at the same time I think there is a possibility to condense the RC and BN rules and rather sort them in some range of importance for BNs to use when scanning a map for game breaking elements etc.
  2. Warn bns that they’re in danger of being put on probation so it doesn’t come across as a surprise

    I got this idea from somewhere else, I forget where. Approaching the date of your revaluation results, there’s a lot of uncertainty as to whether or not you’re doing everything correctly, which in a way makes the revaluation feel like a doomsday. It would be a lot more helpful to have some final warning that you need to improve something before you get surprise-attacked by probation status.

    with the prior considered, I don't this would need any implementation since it would be "more clear" whether or not you are doing well. Also the added workload is rather major in comparison to the affect on BNs being rather minor especially if the previous suggestion is implemented.
  3. Give detailed information as to why each bn is being probationed

    How can you know how to pass probation if you don’t know what to improve? I mentioned earlier that probation isn’t one-size-fits-all but due to the vague process, that’s how it feels like it’s being treated right now. Bns will likely perform better in probation if they know what they did wrong beforehand, and know what they can further improve by the time they’re evaluated again. To compliment this, bns should be able to openly discuss their evaluations with QAT / management person on the subject of their evaluations. (If evaluations happen one month from the evaluation announcement, this can happen like a week or two before the evaluations happen).

    this is condensed in the first suggestion + my response.
  4. Consider bubble pops less than dqs in evaluations

    By weighting bubble pops as evenly as dqs, I don’t feel like we’re thinking about the reason why we have a 2 nominator system in place. In a two nominator system, the two different bns’ perspectives basically ensure that each map is reviewed as thoroughly as possible; if bn #1 doesn’t find an issue with a set, by the time it gets qualified bn #2 is more than likely to have found that issue, because two bns are likely to have different mindsets and processes when modding a map. No map will be perfect after the first bn mod, and dqs are generally seen as a "bigger deal" in most peoples' eyes anyways.
    Additionally, nominations are reset pretty inconsistently between a number of different issues, especially when it comes to issues such as missnaps, tags, or spreads, so it would add a tiny bit more fairness to the process.

    In a sheer statistical sense they already are, 1 BN is less than 2. But Just pulling straight statistic into it, its not really an evaluation to which degree of importance the bubble pop occurred. this would also be condensed with the first suggestion.
  5. Don’t consider dqs or pops for complex timing in evaluations

    Or at least consider them less. Complex timing is not an easy to come by skill that shouldn’t be realistically expected for every bn. Usually when you’re told to ask someone about timing on a map, you hear the same 5 or so names thrown around. I’ve even heard them described as “scary” to nominate because nowadays timing dqs are almost inevitable once your map gets qualified.
    To be clear, simple things like offset and blatantly wrong timing should be taken into consideration. Timing often gets dqed for pretty trivial, small improvements however which makes these maps less attractive to bns.

    funny haha, I agree with Naxess on this one. A rankable consideration is whether or not a beatmap is timed properly. So it should have some value. I think there is a huge oversight in this though. As Naxess stated QATs already don't consider timing in some instances. So there are many BNs who, being the forefront of Ranking Criteria promoters who cannot time a map properly. This is pretty unfair for BNs who are much more experienced in timing. But another thing is that, there is a small minority of people who would be considered reputable in timing altogether. And frankly only a few belong to BNs, and less to QAT. Also the way the current system is set up is doubly unfair for more skilled BNs in terms of timing because, if you miss one spot and your map still gets DQ'd then its rather similar to any sort of DQ for timing (Other than the obviously stated 30ms off). Since timing is so uncomfortable for so many people there should be a reward system instead of a punish system in place for this. This way BNs are more encouraged to show off their timing skills and people who normally wouldn't time maps would start to try and learn it/get comfortable with it. As to how such a thing would exist I can't comment yet but I think this topic in general is unrelated to your proposal and rather silly to include. Also generally speaking it would condense under my response to the first point. Fun topic nonetheless.
  6. Add the ability to request extensions ahead of time

    This is kind of an afterthought to what I talked about in most of the post, but I’ve seen a lot of people become worried about their bn status if their on probation while a big event is happening in their life, or recently, if they had to judge for MWC / some other big event (mostly through #absence_notices in the bn server). Real life should always take priority over circle clicking. It would be a lot more relieving if you were able to tell a QAT / management person that low activity was to be expected, and request be evaluated next month instead.
    Of course if this is a thing, I feel like more limitations would be needed on that prospective bn.

    yeah, sort of, there should be some sort of implementation this but, I wouldn't consider a extension, I would rather see accounts on hiatus from stuff in general (which would overlap motivating the person to not procrastinate in nominating stuff xd) It would also communicate to other people that this person literally CANNOT take requests and what not, and I think it should be completely separate from probation.
Topic Starter
squirrelpascals

Naxess wrote:

Think a lot of this will come after the rework since who knows what'll break or no longer function properly by the time that rolls around with all the changes that are coming. There's the possibility that this whole thing will be replaced by another bn score system as well, so digging too deep into it currently, with the little transparency we have into the coming changes and how/when they'll be accomplished, is probably not a great idea. However, we can still discuss it until then (which should be soon, from the looks of things).


You're right that there's the possibility of the probation system just straight up not existing anymore, but i beleive this is still worth talking about in case it still does by the time the changes happen. I think this system has potential but just needs to be expanded upon for it to work

Write a set of “probation guidelines” that cover what the QAT will use to evaluate you


@Naxess I pretty much agree with you. Case-by-case means that it will have to be able to apply to everybody and examples can be a good guide for those on probation

@bor I feel like everything that wouldn't be able to be covered by guidelines would just fall under common sense. Sure if a bn fails probation they can place the blame on the qat and be wrong, in the end though guidelines are going to give new bns that direction they need, entering the bn world for the first time is a little daunting

Warn bns that they’re in danger of being put on probation so it doesn’t come across as a surprise


@Naxess and bor Yeah I can see where this is coming from, probably easier said than done with the added workload. Maybe with the implementation of the scoring system, assuming that everything will be kept track of on a spreadsheet, this will be a lot easier to gauge? instead of manually having to go to everybody's profiles

Give detailed information as to why each bn is being probationed


@Naxess I think that the opportunity to discuss evaluations the the QAT should be more emphasized then. the "be more careful in general" mindset is whats leaving a lot of probationary bns in the dark, and if it's true that those people are going to have an easier time passing then it's just unneeded stress. Since bn statuses are on the line the goal should be to provide as much closure as possible

Consider bubble pops less than dqs in evaluations


@Naxess Pretty much everything you said makes sense. The reason I brought it up is because again, there are issues that are inconsistently popped / dqed for, and in more debatable cases this would probably incite less fear in being the first nominator on riskier maps where there is more uncertainty in certain issues. The discussion between the two nominators would help to fix the problem, and if a bubble pop is part of that process then again it's only a natural part of the system.

Not exactly a huge problem but I thought it was worth mentioning on the topic. This goes hand in hand with what @bor said if both bns are wrong too :p

Don’t consider dqs or pops for complex timing in evaluations


@Naxess Again though, only a small proportion of bns can probably consistnetly spot these issues if they checked a bunch of maps with complex timing in a row, because it's so precise and its difficult to come up with solutions that everyone can agree with in many cases. I feel like currently the proportion of the number of maps with complex timing is very uneven with the number of bns who are sufficient enough in timing to check them with confidence they won't get a dq. If something about weighting timing less (vs. not at all) were to be noted somewhere that would be relieving to a lot of people.

@bor The thought of a reward system is really creative :D I was thinking what if a "league" of people who were sufficient in timing existed, kind of like the mappers guild of timing. thats not relevant to the proposal at all but i can see a rewards system being relevant within that.

Add the ability to request extensions ahead of time

@Naxess My thought here was to make the option openly available for anyone who konws they will probably land on probation for activity. So pretty much what you guys are currently doing but just saying "this is an option if something important happens."

@bor So like, getting rid of that person's bn status entirely is the limitation? I guess I can kind of agree to that. If the "limitation" is not being able to nominate stuff I don't know if there's another way lol
anna apple

squirrelpascals wrote:

Write a set of “probation guidelines” that cover what the QAT will use to evaluate you
@bor I feel like everything that wouldn't be able to be covered by guidelines would just fall under common sense. Sure if a bn fails probation they can place the blame on the qat and be wrong, in the end though guidelines are going to give new bns that direction they need, entering the bn world for the first time is a little daunting in a system that is based on a more hand holding type style there is no "common sense" since that obviously will differ from a new bn Versus and experienced qat.

Warn bns that they’re in danger of being put on probation so it doesn’t come across as a surprise
@Naxess and bor Yeah I can see where this is coming from, probably easier said than done with the added workload. Maybe with the implementation of the scoring system, assuming that everything will be kept track of on a spreadsheet, this will be a lot easier to gauge? instead of manually having to go to everybody's profiles

its "easier" under the assumption that the QAT are not also human and need periods of time to do any non QAT related work

Add the ability to request extensions ahead of time

@bor So like, getting rid of that person's bn status entirely is the limitation? I guess I can kind of agree to that. If the "limitation" is not being able to nominate stuff I don't know if there's another way lol my thing is not to "get rid of them" more like just lock them out of it for a while, when they have their lives together again the effort they spent getting in the first time is testament to their skill (at least to some degree) enough to grant them "immediate" BN membership
pishifat
considering the qat/bn system is being revamped and this thread is inactive, gonna archive
Please sign in to reply.

New reply