forum

Generative AI for automatic beatmap creation (BETA AVAILABLE)

posted
Total Posts
13
Topic Starter
sedthh
Hi everyone, I've been working on a Neural Network model that takes any mp3 as input and turns it into beatmaps for rhythm games. I have started with OSU as it is one of my favorites, but the long term goal is to support all kinds of different platforms:
- Beatmaps are converted to an intermediate file format, which is just an excel sheet containing "tracks" with timing. Only beat and hold events are encoded (OSU maps have 1 track, MANIA tracks have 4).
- Then this intermediate file format is tokenized (turned into numbers the model can understand), and is fed into the network combined with the audio.
- The model is then trained to predict the next token (a series of beatmap events) based on both previous events and the surrounding audio.

I am currently in the process of training the first models and making them publicly available:
https://github.com/sedthh/BeatLearning
Lp9
do you have examples of already generated maps?
Topic Starter
sedthh
My plan was to release an example beatmap today alongside with the library, but it takes way longer to train the first model than I have thought (I'm still training the model since yesterday). The first versions are still a bit off, but it clearly just needs more data to improve. I will share an example right away once I feel like the output is decent!

NOTE: this could take anywhere from a few days to a week or two, but I wanted to share it anyway because I need support with the beatmap conversions
abraker
What gpu are you using to train it?
Topic Starter
sedthh
I have a 4090, the model itself is small though (I wanted to keep both the context window and the voacbulary size for the tokenized hit events small, due of the lack of training data available).
Octostomp
This seems interesting! Do you have to set CS, Accuracy/OD, and all that stuff, or does it set it based off the song? Also, does it have options to set difficulty? If it can make rankable 1 stars, I might finally have something to play lol
Topic Starter
sedthh

Octostomp wrote:

This seems interesting! Do you have to set CS, Accuracy/OD, and all that stuff, or does it set it based off the song? Also, does it have options to set difficulty?
You just need to upload an MP3, select a difficulty and let the model generate the beatmap.
But you will still need to play around with sampling settings to get better results, just like with any gen AI.

Right now it still sometimes misses timing by a bit, sometimes leaves in silent parts but sometimes does a really good job when it is able to pick up some motive.

Octostomp wrote:

If it can make rankable 1 stars, I might finally have something to play lol
That would actually be possible but that's where I will need the most help from the community: to convert the models output to beatmap formats and add metadata so that they actually fit the quality criteria.
Topic Starter
sedthh
Hey, I am done with the first release, can someone suggest me a few copyright-free songs that I can use to create examples with? I will share the converted beatmaps.

Meanwhile you can try it out on your own via https://github.com/sedthh/BeatLearning if you have a decent GPU (or Google Collab credits)
Nao Tomori
You can use any featured artist song (beatmaps/artists)
Topic Starter
sedthh

Nao Tomori wrote:

You can use any featured artist song (beatmaps/artists)
Thanks! I will look into it

UPDATE:
You Are Mine - S3RL Ft Kayliana
https://www.youtube.com/watch?v=tyhkMdpbRi8
Topic Starter
sedthh
fireseal
Is this project still being developed? This sounds very cool and I wonder how you would train the AI for harder difficulties unless the process is mostly the same
play dthr
ive seen something similar works up to 7* nm

https://www.reddit.com/r/osugame/s/HqzGmiXuzP
Please sign in to reply.

New reply