Guest
home
beatmaps
rankings
community
store
help
forum
  1. Forums
  2. osu!
  3. Development

fighting cheating in Osu! with AI anti-cheat

posted
Total Posts
11
hyperastro
8 posts
Joined May 2023
Topic Starter
hyperastro
I’m really tired of cheaters, and I think we all are (unless you're a cheater, lol). So, I decided to start developing an Osu AI anti-cheat system similar to VAC-net in CS2.

I'm trying to collect as many cheated and non-cheated replays as possible to train an AI model (BERT in this case because of its bi-directional tokenization) to detect, with a high degree of accuracy, whether a replay or score is cheated.
This AI will be trained to detect not only top players cheating but also the lower-ranked players who set impressive scores that go unnoticed. To achieve this, I need a lot of replay data from both high and low-level players.

My goal for this project is to have a AI model scan over replays from maps and then notifying the Osu! team of possible cheated scores to help with catching cheaters

if you want to join the project join our Discord
https://discord.gg/BDU3W22HEW

or our github
https://github.com/160IQAstro/Osu-AI-Anti-Cheat-Project/tree/main

P.S. : This is still very early in development so don't expect to see much on the github as of yet but for anyone who wants to submit replays please join the discord
Last edited by hyperastro , edited 1 time in total.
Purple Jaguar
1 post
Joined November 2020
Purple Jaguar
The discord link for me is invalid. Looks like a cool project that i'd be willing to help out with after I finish some things.
Last edited by Purple Jaguar , edited 1 time in total.
hyperastro
8 posts
Joined May 2023
Topic Starter
hyperastro

Purple Jaguar wrote:

The discord link for me is invalid. Looks like a cool project that i'd be willing to help out with after I finish some things.
fixed the link, Thank you for your interest!
anaxii
15,000 posts
Joined December 2022
anaxii
The only thing I'm scared of is the AI detecting a cheater that isn't even one, like an offline player or something similar.
hyperastro
8 posts
Joined May 2023
Topic Starter
hyperastro

I AM VERY SMART wrote:

The only thing I'm scared of is the AI detecting a cheater that isn't even one, like an offline player or something similar.
Although I believe that this Model will be able to detect cheats with a high degree of accuracy as you said there is always the possibility of a false positive, and I think the final decision to ban should always be made by a human. I see the role of this neural network to aid and automate the catching of cheaters.
Winnyace
633 posts
Joined June 2016
Winnyace
Are some interesting nuggets for you about VACNet:
  1. VACNet required a lot of data and it was funneled through the Overwatch system because AI can't always accurately detect cheats or not, thus Valve decided that instead of letting the AI hallucinate, they decided to let players do some of the work and verify
  2. VACNet, in CSGO, didn't progress past detecting spinbots
  3. To my knowledge, CS2 still has a large cheating problem and while there are improvements with bans being delivered, cheaters are still at it
  4. There are rumors that cheaters made a bot to make Overwatch cases, then through an exploit, gain the data from the anonymized replays in order to save the skin of their customers (since in CS and other big games, this is a serious business and outlet to make money)
The amount of data required for this AI is very large and it needs to be sifted through. AI isn't ready for any new cheating methods innately. You would need to make sure this data is correct and accurate and kept updated with new behavior and cheats. You would need to find a way to constantly retrain bits and pieces of it in order to not hallucinate and ban or detect players that aren't cheating too.

Thus, I frankly believe this is going nowhere. I'm sorry, but I don't see future in this, at the moment.
Last edited by Winnyace , edited 3 times in total.
hyperastro
8 posts
Joined May 2023
Topic Starter
hyperastro

Winnyace wrote:

Are some interesting nuggets for you about VACNet:
  1. VACNet required a lot of data and it was funneled through the Overwatch system because AI can't always accurately detect cheats or not, thus Valve decided that instead of letting the AI hallucinate, they decided to let players do some of the work and verify
  2. VACNet, in CSGO, didn't progress past detecting spinbots
  3. To my knowledge, CS2 still has a large cheating problem and while there are improvements with bans being delivered, cheaters are still at it
  4. There are rumors that cheaters made a bot to make Overwatch cases, then through an exploit, gain the data from the anonymized replays in order to save the skin of their customers (since in CS and other big games, this is a serious business and outlet to make money)
The amount of data required for this AI is very large and it needs to be sifted through. AI isn't ready for any new cheating methods innately. You would need to make sure this data is correct and accurate and kept updated with new behavior and cheats. You would need to find a way to constantly retrain bits and pieces of it in order to not hallucinate and ban or detect players that aren't cheating too.

Thus, I frankly believe this is going nowhere. I'm sorry, but I don't see future in this, at the moment.


It is true that CS has currently a lot of cheaters, even with VAC-NET, but that is because Valve is being very careful with how they moderate VAC-NET. There have been several other AI networks trained by the CS community that have had "better" detection rates than VAC-NET, most notably HestiaNET. In terms of training data, I have managed to curate 20k replays that are non-cheated, but only a couple of thousand that are cheated. Although I am still not satisfied with the amount of replays I have collected, I am currently talking to different people who host replay mirrors to send me all the replays on their servers, which I will then filter through using various techniques. Also, I DO NOT INTEND FOR THIS AI MODEL TO REPLACE MANUAL REVIEW OF REPLAYS AND JUDGEMENT on whether they are cheated or not. I just want to develop a tool that will help and make the process of finding these cheated replays more efficient. Thank you for sharing your concerns, I like to listen to different perspectives.
If you have any more concerns don't hesitate to join the Discord or DM me personally.People who provide constructive criticism like yourself are one of the most important people to a project as they let me check my biases and previous judgements!
Winnyace
633 posts
Joined June 2016
Winnyace

hyperastro wrote:

I am currently talking to different people who host replay mirrors to send me all the replays on their servers, which I will then filter through using various techniques.
That's a lot of data that needs to be categorized. I am not sure what techniques you could to do this accurately, besides getting a lot of people to do that manually alongside you. I believe any automated techniques can add errors by mistake, thus making the model for the AI wrong, thus leading to it making the wrong decisions. Humans can do that too, however... I don't know.
hyperastro
8 posts
Joined May 2023
Topic Starter
hyperastro

Winnyace wrote:

hyperastro wrote:

I am currently talking to different people who host replay mirrors to send me all the replays on their servers, which I will then filter through using various techniques.
That's a lot of data that needs to be categorized. I am not sure what techniques you could to do this accurately, besides getting a lot of people to do that manually alongside you. I believe any automated techniques can add errors by mistake, thus making the model for the AI wrong, thus leading to it making the wrong decisions. Humans can do that too, however... I don't know.
We are going to train the neural net on replays that were filtered by circle-guard(a tool made to detection osu! cheated replays).After its been trained we will fine-tune the model with manually reviewed replays to further improve its accuracy. We will iterate this process to keep it updated with new-cheating methods.



Phaeth0n
Phaeth0n

hyperastro wrote:

I’m really tired of cheaters, and I think we all are (unless you're a cheater, lol). So, I decided to start developing an Osu AI anti-cheat system similar to VAC-net in CS2.

I'm trying to collect as many cheated and non-cheated replays as possible to train an AI model (BERT in this case because of its bi-directional tokenization) to detect, with a high degree of accuracy, whether a replay or score is cheated.
This AI will be trained to detect not only top players cheating but also the lower-ranked players who set impressive scores that go unnoticed. To achieve this, I need a lot of replay data from both high and low-level players.

My goal for this project is to have a AI model scan over replays from maps and then notifying the Osu! team of possible cheated scores to help with catching cheaters

if you want to join the project join our Discord
https://discord.gg/BDU3W22HEW

or our github
https://github.com/160IQAstro/Osu-AI-Anti-Cheat-Project/tree/main

P.S. : This is still very early in development so don't expect to see much on the github as of yet but for anyone who wants to submit replays please join the discord

hyperastro wrote:

I’m really tired of cheaters, and I think we all are (unless you're a cheater, lol). So, I decided to start developing an Osu AI anti-cheat system similar to VAC-net in CS2.

I'm trying to collect as many cheated and non-cheated replays as possible to train an AI model (BERT in this case because of its bi-directional tokenization) to detect, with a high degree of accuracy, whether a replay or score is cheated.
This AI will be trained to detect not only top players cheating but also the lower-ranked players who set impressive scores that go unnoticed. To achieve this, I need a lot of replay data from both high and low-level players.

My goal for this project is to have a AI model scan over replays from maps and then notifying the Osu! team of possible cheated scores to help with catching cheaters

if you want to join the project join our Discord
https://discord.gg/BDU3W22HEW

or our github
https://github.com/160IQAstro/Osu-AI-Anti-Cheat-Project/tree/main

P.S. : This is still very early in development so don't expect to see much on the github as of yet but for anyone who wants to submit replays please join the discord
would that be a little to dangerous since it is AI it needs a starting point or it could go horribly wrong and accidently do a ban wave
hyperastro
8 posts
Joined May 2023
Topic Starter
hyperastro

Phaeth0n wrote:

would that be a little to dangerous since it is AI it needs a starting point or it could go horribly wrong and accidently do a ban wave
its a valid concern that an AI anti-cheat could cause a ban wave, but as I mentioned before I believe that the final judgement should always be made by a human and the AI should not be able to ban people but only notify moderators. So it is unlikely that something like that happens!(at least with this project)
Please sign in to reply.

New reply

0
1 / 11