Guest
home
beatmaps
rankings
community
store
help
forum
  1. Forums
  2. osu!
  3. Development

Generative AI for automatic beatmap creation (BETA AVAILABLE)

posted
Total Posts
17
sedthh
9 posts
Joined May 2008
Topic Starter
sedthh
Hi everyone, I've been working on a Neural Network model that takes any mp3 as input and turns it into beatmaps for rhythm games. I have started with OSU as it is one of my favorites, but the long term goal is to support all kinds of different platforms:
- Beatmaps are converted to an intermediate file format, which is just an excel sheet containing "tracks" with timing. Only beat and hold events are encoded (OSU maps have 1 track, MANIA tracks have 4).
- Then this intermediate file format is tokenized (turned into numbers the model can understand), and is fed into the network combined with the audio.
- The model is then trained to predict the next token (a series of beatmap events) based on both previous events and the surrounding audio.

I am currently in the process of training the first models and making them publicly available:
https://github.com/sedthh/BeatLearning
Last edited by sedthh , edited 1 time in total.
Lp9
253 posts
Joined May 2020
Lp9
do you have examples of already generated maps?
Last edited by Lp9 , edited 1 time in total.
sedthh
9 posts
Joined May 2008
Topic Starter
sedthh
My plan was to release an example beatmap today alongside with the library, but it takes way longer to train the first model than I have thought (I'm still training the model since yesterday). The first versions are still a bit off, but it clearly just needs more data to improve. I will share an example right away once I feel like the output is decent!

NOTE: this could take anywhere from a few days to a week or two, but I wanted to share it anyway because I need support with the beatmap conversions
Last edited by sedthh , edited 1 time in total.
abraker
Global Moderator
8,327 posts
Joined July 2014
abraker
What gpu are you using to train it?
sedthh
9 posts
Joined May 2008
Topic Starter
sedthh
I have a 4090, the model itself is small though (I wanted to keep both the context window and the voacbulary size for the tokenized hit events small, due of the lack of training data available).
Octostomp
55 posts
Joined February 2024
Octostomp
This seems interesting! Do you have to set CS, Accuracy/OD, and all that stuff, or does it set it based off the song? Also, does it have options to set difficulty? If it can make rankable 1 stars, I might finally have something to play lol
sedthh
9 posts
Joined May 2008
Topic Starter
sedthh

Octostomp wrote:

This seems interesting! Do you have to set CS, Accuracy/OD, and all that stuff, or does it set it based off the song? Also, does it have options to set difficulty?
You just need to upload an MP3, select a difficulty and let the model generate the beatmap.
But you will still need to play around with sampling settings to get better results, just like with any gen AI.

Right now it still sometimes misses timing by a bit, sometimes leaves in silent parts but sometimes does a really good job when it is able to pick up some motive.

Octostomp wrote:

If it can make rankable 1 stars, I might finally have something to play lol
That would actually be possible but that's where I will need the most help from the community: to convert the models output to beatmap formats and add metadata so that they actually fit the quality criteria.
sedthh
9 posts
Joined May 2008
Topic Starter
sedthh
Hey, I am done with the first release, can someone suggest me a few copyright-free songs that I can use to create examples with? I will share the converted beatmaps.

Meanwhile you can try it out on your own via https://github.com/sedthh/BeatLearning if you have a decent GPU (or Google Collab credits)
Last edited by sedthh , edited 4 times in total.
Nao Tomori
osu! Alumni
3,106 posts
Joined December 2014
Nao Tomori
You can use any featured artist song (beatmaps/artists)
sedthh
9 posts
Joined May 2008
Topic Starter
sedthh

Nao Tomori wrote:

You can use any featured artist song (beatmaps/artists)
Thanks! I will look into it

UPDATE:
You Are Mine - S3RL Ft Kayliana
https://www.youtube.com/watch?v=tyhkMdpbRi8
Last edited by sedthh , edited 1 time in total.
sedthh
9 posts
Joined May 2008
Topic Starter
sedthh
Here's another one:
https://www.youtube.com/watch?v=KYU2QJx4_yQ
fireseal
7 posts
Joined January 2016
fireseal
Is this project still being developed? This sounds very cool and I wonder how you would train the AI for harder difficulties unless the process is mostly the same
troIIing
181 posts
Joined November 2018
troIIing
ive seen something similar works up to 7* nm

https://www.reddit.com/r/osugame/s/HqzGmiXuzP
sedthh
9 posts
Joined May 2008
Topic Starter
sedthh

troIIing wrote:

ive seen something similar works up to 7* nm

https://www.reddit.com/r/osugame/s/HqzGmiXuzP
Oh my god, thanks for posting this! I just contacted the person. His model makes beatmaps from timings, while mine makes timings from music. So if we combined them, we could create something spectecular!


fireseal wrote:

Is this project still being developed? This sounds very cool and I wonder how you would train the AI for harder difficulties unless the process is mostly the same
I am going to rewrite the whole thing during the holidays and create an updated model.
eatcake
27 posts
Joined June 2018
eatcake
umm... can you maybe like not give it to the public

if it works well i mean
Last edited by eatcake , edited 1 time in total.
ARNABH_OP
11 posts
Joined March 2023
ARNABH_OP
<Hey There!>
Thank you for working on such an amazing project! Turning mp3s into beatmaps using AI is an incredible idea, and it’s exciting to see someone bringing it to life. Your efforts are truly inspiring—keep it up! 🚀🎶
eatcake
27 posts
Joined June 2018
eatcake

ARNABH_OP wrote:

<Hey There!>
Thank you for working on such an amazing project! Turning mp3s into beatmaps using AI is an incredible idea, and it’s exciting to see someone bringing it to life. Your efforts are truly inspiring—keep it up! 🚀🎶
ai response 😭
Please sign in to reply.

New reply

0
1 / 17