Disclaimer: The term of "content reviewing" maps will pop up a few times here. If you want to read up on how it's performed, click here


Hey, writing this as another map fell victim to the inconsistent and mysterious content moderation system used today - at first, the background was outright banned without engaging with nor providing any sort of insight into the responsible people's discussion to the community for including minor nudity, even though theoritically such cases for artistic calls are allowed per Visual Content Considerations. It was retracted upon and put up for content review instead, which is still commendable, but it signals that something needs to change if similar conflicts in this regard keep happening repeatedly

To be specific, I believe the main issue with the current way of executing content rules is the fact that more often than not, we don't get to see any information on why something was considered (in)appropriate besides "read VCC and apply to your heart's content", in which case even map hosts are forced to ask for more information on their own, unless they'd issued the discussion themselves (via content reviewing for example). This also makes it easy to just apply your personal views by voting "Yes"/"No" without actually backing it up because the community won't be able to see it 99% of the time, which removes any sort of accountability on top of that


For perspective, let's see vetoes - DQ's on issues subjectively deemed unrankable by full BN's/NAT's (sounds alike as it only takes one person to report a map and temporarily take it down to hold a vote, except it's limited here), treated in a similar way through votings if consensus cannot be reached. The elephant in the room in comparing both is that all vetoes have a dedicated page which is easily found here, including reasons backing up the point of view of each voter on the subject, eg. the infamous Spelunker veto. The additional positive here is that if a vote explanation comes off as ignorant, contradicting generally agreed upon facts etc., it can be pointed out and held responsibility for if needed

I'm concerned why a system like this hasn't been implemented for visual content moderation in maps yet as it already proved to have been able to spark similarly massive discussions in the past (some of them you may find familiar if you are here long enough), a few prominent examples being: [DISCUSSION 1] [DISCUSSION 2] [DISCUSSION 3]; All three of them, the discussion linked in the 1st paragraph and the system at hand share the aforementioned problem. Up to this day, it mostly derails to either:

1. "Hey, content you used has been internally deemed inappropriate per VCC and should be removed"
2. "Hey, content you used has been reviewed and can(not) be used, here are the numbers: *percentages*"

While I know neither will ever be perfect due to the inconsistency of the human factor used, I believe enforcing VCC/putting up content reviews on maps could benefit a lot from taking after the veto system. It'd seem a lot more clear and bearable regardless of the outcomes if these two changes were applied:

1. A page for reasons/results/detailed votes on each content review case held (and potentially summaries on internal content-related discussions within the BNG/GMT) was created on the BN site, similar to how vetoes work - laying out all information needed on the table to the community would provide a lot more transparency on the subject. The latter seem scarce in comparison so alternatively, including those summaries in github threads and keeping the subpage exclusive to content reviews could also work, using github this way was already attempted in the Spelunker veto conclusion
2. Justifications for content review votes were required, at least for "No" votes - you should be able to explain why you consider something unacceptable and, in this case, direct to the proper rule/VCC bullet. Because it's safe to assume content reviews are issued a lot more frequently than vetoes and the problematic field is limited, it could be kept very concise - briefly outlining why content in question is (not) okay to let through in one's opinion likely won't take more than 1-2 sentences

There is more that could be added to the list but I believe this covers enough, will gladly take other suggestions for potential discussion/corrections if something'd been discussed/implemented before this was written


ETA: Looking back at some stuff, I found that this was a thing, may be worthwile to include in the discussion

fully supporting this

Providing pages for each contents reviewed is a good idea. However, here is the problem; how does the page presents these accepted and unaccepted (which fell into VCC violation) contents? Some are NSFW and violate this website's policy, either.

Giving the link to the content without thumbnailing it might be the workaround, howveer links don't last forever. There should be a way to maintain and preserve even VCC-violating images to maintain the transparency.

The idea is great, though. I support this though there are some things need to be considered here and there.

Just for some historical context, the current content review system was developed as a compromise solution to the old model of "one-off" or "closed" visual content enforcement, in part because said model was producing large veto-like discussions and huge amounts of animosity between people participating in these discussions when their viewpoints did not align.

Put shortly, osu! is a game with a global community, combining massively varied diasporas of peoples, cultures and expectations into one space - especially regarding visual content and what is acceptable. The current system is designed to incorporate as much organic feedback from the GMT and the BN teams as possible in the hopes that the best result is provided by numerical consensus, that is, the best result 'comes out in the wash'. Any system that is made for this kind of thing is always going to face some form of heavy criticism no matter what changes are made because of this. I feel this is important to get out of the way early on for reasons that should become apparent later on.

In regards to your listed points:

A page for reasons/results/detailed votes on each content review case held

This should be fine and is largely possible, but will not consist of anything more than the per-group percentages and maybe the raw yes/no numbers. I'm not entirely sure this will do much to help people feel more comfortable with the system, but I also don't see any harm in making it publicly visible if it isn't already. Exposing individual user votes is unlikely to happen - participation in the content review process is relatively sparse as is, and doing this will deter people from engaging with the system to begin with, which only weakens it further.

Require justifications for content review votes

I have mixed feelings about this. I see where you're coming from with it and agree at a base level that some kind of justification should be required, but the presence of said justification also infers that the justifications then need to be judged. This then begs the question: who is going to judge whether a justification is reasonable or not?

If it is the greater community at large, participation in the content review process will suffer, as above. GMT members are considered on equal footing with one another and thus cannot readily discard each other's opinions - do we then vote on individual justifications for every contentious content review process? It takes several days and sometimes weeks for a GMT vote to pass, so this is not really feasible.

The problem with public justifications is that if they're written, people will be able to identify others based on writing style. We could potentially trial a checklist type thing where CR participants are required to tick off which VCC rules they think are being broken by a piece when no-voting, but this is still only really going to address half the problem as maps that are given a pass when technically breaking the VCC are by nature, not going to be required to fill out this checklist.

The current reality of the content review process is that most BN and GMT click into the CR card on the bnsite and basically reflexively and intuitively vote on whether they think something is allowed or not. Officially, people are supposed to take a few moments to compare the content up for vote with the VCC and act with those rules in mind, but they generally do not, and this is something I have raised repeatedly with the team to no obvious result.

Things definitely require improving but it is not precisely clear which direction should be taken at the moment.

Ryu Sei wrote:

Providing pages for each contents reviewed is a good idea. However, here is the problem; how does the page presents these accepted and unaccepted (which fell into VCC violation) contents? Some are NSFW and violate this website's policy, either.

Giving the link to the content without thumbnailing it might be the workaround, howveer links don't last forever. There should be a way to maintain and preserve even VCC-violating images to maintain the transparency.

The idea is great, though. I support this though there are some things need to be considered here and there.

It's common knowledge that age verification in the internet isn't possible without case-by-case ID photo profiling. Something that could be done to at least warn the user before proceeding is prompt a warning window: "The site you are about to open may contain potentially NSFW content, are you sure you want to keep moving forward?". Although not much, it's always better than nothing

This might not even be troublesome though, given that BN's don't need to be verified 18+ users despite needing the ability to view and vote on visual stuff

Something like the current BN evaluation system could work, we assign a pool of GMT (NAT + BNas well?) members and they are given a week or so to vote and provide some commentary as to why they have chosen to vote this way. Afterwards a short summary is written and a content review link is available to everyone.

Pretty sure eph has already explained the rest of the points regarding consistency and deanonimization, so I'll just not comment on that.

I generally agree with Dabbe_01's proposal.

I believe that all members of the GMT/NAT (and BNG in case of a non-70% 1st round) should get a vote in a Content Review as opposed to a jury of 11-ish people, because of what Ephemeral said ("osu! [has] a global community..."). A high sample size is a good idea.

I also think filling out a checklist should be required, especially for 'no' votes, and a tally of reasons for votes should be presented alongside the result. This would:

1. force the reviewers to analyze the image beyond their initial 'gut sightread',
2. make it clear why the reviewers oppose the use of a visual element,
3. an optional written reason may be given to accompany a checked list point.
   
   But how is anonymity maintained with written reasons?

   To maintain anonymity, the written reason(s) will go through a proofreader who will remove obvious writing style markers (e.g. tomatas95 and their use of tildes~) if they are sure they are errors/style markers as opposed to grammatically relevant. The proofreader will not know who wrote the original reason, and are prohibited from sharing the original reason text or their guess as to who wrote the reason.
   (The writer of the reason will be informed of what their reason looks like once edited, and may request another proofreader once if they feel the 1st one is biased/misrepresentative.) Who this proofreader should be, I'm not sure; maybe a GMT/NAT member with the same vote as the writer of the reason.

A page with the result of votes should be provided, linked alongside the post challenging the content in the map mod thread. This page should show the number of votes, percentage of votes, amount of checks on specific rules, and the specific justification as to why each rule is considered 'broken'.

@Ephemeral sry for the hassle

Ephemeral wrote:

This should be fine and is largely possible, but will not consist of anything more than the per-group percentages and maybe the raw yes/no numbers. I'm not entirely sure this will do much to help people feel more comfortable with the system, but I also don't see any harm in making it publicly visible if it isn't already. Exposing individual user votes is unlikely to happen - participation in the content review process is relatively sparse as is, and doing this will deter people from engaging with the system to begin with, which only weakens it further.

Although raw numbers won't contribute to the issue significantly, it's a step in the right direction and I'm glad to know that it may be implemented without problems. As for exposing individual votes - I still believe that this is both crucial to aid the system in the long run and not that problematic at first glance if BN's know what they're doing. As for BN's (not) willing to participate in the new votings, that's a whole new can of worms I'm not certain ought to be open just yet, although additives could be (depending whether you want to play the good or bad cop) either minimum activity requirements for votings, or extra incentives like a bit more leniency on DQ's for more outstanding voters

Ephemeral wrote:

If it is the greater community at large, participation in the content review process will suffer, as above. GMT members are considered on equal footing with one another and thus cannot readily discard each other's opinions - do we then vote on individual justifications for every contentious content review process? It takes several days and sometimes weeks for a GMT vote to pass, so this is not really feasible.

I'd had a different reason for this in mind than having every justification undergo outside judgement immediately. The initial thought was that if such reasonings were given along with the team's consensus, it'd highlight that they truly know what they're doing as it's vital for being able to back up your perspective, which should also result in fewer potential slip-ups with abrupt disqualifications that cause the biggest public uproar

Ephemeral wrote:

The problem with public justifications is that if they're written, people will be able to identify others based on writing style. We could potentially trial a checklist type thing where CR participants are required to tick off which VCC rules they think are being broken by a piece when no-voting, but this is still only really going to address half the problem as maps that are given a pass when technically breaking the VCC are by nature, not going to be required to fill out this checklist.

Isn't this already a thing with vetoes though? People are able (and expected) to write down their thoughts in detail for the sake of veto mediations, which is what reviewing visual content is partially based on already. With significantly larger pools, it'd be more difficult to identify specific people via their responses if it's not given away from the get-go. Furthermore, the hate would be less concentrated on specific people even in case they actually got identified (they're a much smaller part of a group in proportion here after all)

As for the checklist proposal - I see this working, at least as a trial solution. Even when looking at VCAP which I linked in the ETA (which I'd consider good to refer to despite having found it in a 2-year-old discussion), a statement can be found that: "Something that ticks off a lot of the check boxes (3 or more is a good indicator of something worth further thought) in here is likely to be problematic [...]". This seems perfectly applicable here, possibly with built-in remarks that a checklist has multiple problems marked, especially if this occurence happens for multiple voters. Adding a way to briefly explain how each box marked is relevant to the assessed content could improve the solution even further. For now though, picking options from a limited set appears to be a quicker and more comfortable way of enforcing a vote justification and may be not too deterrent compared to the necessity to write text yourself (especially for non-fluent English speakers)

Ephemeral wrote:

The current reality of the content review process is that most BN and GMT click into the CR card on the bnsite and basically reflexively and intuitively vote on whether they think something is allowed or not. Officially, people are supposed to take a few moments to compare the content up for vote with the VCC and act with those rules in mind, but they generally do not, and this is something I have raised repeatedly with the team to no obvious result.

Can't really blame people for this, everyone attempts to take the easiest route (in this case, it is relying on intuition), even though this shouldn't be the case and unfortunately, the system is way too inefficient in filtering out these 100% subjective votes. A proposed checklist should be comfortable enough to at least let both sides meet somewhere in the middle

Also, regarding proofreaders mentioned in another post: fwiw it seems excessive, and trying to automate the process may either not even be close to sufficient (if implemented quickly) or just hinder the attempts for any change

I'm going to just say that I support on having a more streamlined process for everyone to be able to know the reasons for a BG/Media being removed. I think it's fair for me to say that some people believe that the application of these rules/guidelines is at times inconsistent. Having a place to see the outcome of the voting and conclusory comments to determine what caused X to be treated differently than Y for everyone to see is already a good first step.

We already have a page for this though! Just like everyone can see the Veto page, I think that a change that could be done in the short term is doing the same for the results of content reviews, which users cannot currently see at all.

Moreover, I sort of agree that enforcing some sort of justification beyond a Yes/No and for that to be publicly seen is a good solution. Again coming back to the Vetoes, this has already been done. Yes it is true that there might be bad actors that can try to guess who's behind what. But that's not really the point of this? Any modding discussion, sudden DQ, QA mods,... all of these are susceptible to this. I feel like the transparency benefit outweighs the problems that other aspects of modding/ranking/uploading sets already carry over.

If activity is a concern regarding forcing these checks, this can be an opt-in option for BNs, and maybe just like Evals do, have it randomly assigned to a group of GMTs to comment on. I can only speak from the BN side of things obviously, so if there's a GMT that can tell us how this would affect their workflow, that'd be awesome.

^ would be awesome to have a GMT/NAT member describe it from their perspective, especially as it died down another time and we haven't been given one yet

Dabbe_01 wrote:

^ would be awesome to have a GMT/NAT member describe it from their perspective, especially as it died down another time and we haven't been given one yet

I didn't know that transparency on this isn't a thing yet, honestly lol....
There's already pages for each of the votes on the BN site, people probably just can't see them if they're not BN or gmt i guess?

BLAHBLAH BLAH DISCLAIMER THAT THIS IS MY OPINION AND NOT THE "GMT AS A WHOLE"

Here's what I struggle with in regards to VCC reviews stuff:

1. searching for any old BG unless you know that there's been a vote on it sometime in the past is near impossible, if the image link in the puush is dead the vote is useless. If the "cover image" of the beatmap has been used the image in the vote will update to teh changed image making the entire thing useless too...
2. The verdict doesn't include reasoning or transparency as to why it was deemed inappropriate (we just vote and the resulting verdict is kind-of-law)
3. Communicating the verdict in an actionable way is often not thought of because for a bunch of bgs it's just like a matter of "get something else", if someone wants to censor their bg or whatever and has the actual skill in order to pull it off, getting directions is often very tedious
   
   Especially applies for videos
4. voting generally takes long from a process perspective, 3 to 7 days to get a "yes / no" out of a team is more bureaucracy than the resulting call oftentimes feels is worth

Furthermore we have a checklist of what is and isn't allowed or what areas to pay attention to, but it often feels like half the checklist is disregarded by the voting parties regarding posturing/clothing. Might just be me. Stuff where I'd be like wow why's there so much of her butt in this image pass with 100% agree, while stuff where I honestly fail to see a problem gets nuked. Which is what democracy is good for, but IDK if that means we should audit the process as a whole as well

IMO making the results transparent would be cool, would also be cool if the community was less trigger happy to shoot the messenger on delivering the verdict of the vote because they don't necessarily agree with the outcome of the vote either so downvoting them or whatever honestly doesn't achieve anything

100% in favor for at least giving people access to the vote results, and maybe introducing better verdicts for "what would need to change and what area violates VCC"

Following the proposal, the content review website, where you can see results of each content review vote was opened up for every user for now.

I like the direction of this transparency. It's not as much as I demanded but at least now we can validate the results!

The explicit content warning can be more obvious, but that's fine. Users going to see the result of reviews should be conscious that they may view explicit contents.