osu! beatmap generator with Tensorflow / Deep Learning

posted
Total Posts
18
Topic Starter
[CSGA]Ar3sgice
demo map: https://osu.ppy.sh/beatmapsets/834264

code repository: https://github.com/kotritrona/osumapper

Deep Learning model in osu! using Tensorflow.

After training a neural network from some existing maps, it can automatically generate a new map for any given music.



It uses 3 different models for different purposes.

Firstly, it extracts audio feature using FFT (fourier transform) at different levels, combined with timing data, which gets sent to the rhythm model and the momentum model.

The rhythm model has some CNN (convolutional neural network) and LSTM (long-short term memory) layers, modelling note presence and note type at each timing step (1/4s of a beat).

The momentum model has the same structure as the rhythm model, but its input data only contains the notes, not empty air. It models and predicts momentum (distance / time interval) at each note, so the model will know when to jump.

Then, after predicting rhythm and distances for a new map, the data is used to train flow with a GAN (Generative Adversial Network). This will try to map the flow in a similar style of the training mapset.

After training the GAN for some time, all the data is converted to .osu format and output.
First install the dependencies:
  1. Python3 with IPython Notebook. (I use Canopy with Python 3.5)
  2. Tensorflow version 1.9.0 or 1.10.0.
  3. Node.js.
  4. FFmpeg (to convert mp3s to wavs for the program to read)
  5. pip install soundfile
  6. npm install polynomial (to calculate positions of slider ends)


Then, find some maps you want to train with, and put the paths into a maplist.txt file.
(the repository has an example file)

If you have no idea about what map to train, start from the 5th step to use the default model (Sota dataset).

Run the notebooks:
  1. put maplist.txt in place and run 01_osumap_loader.ipynb
  2. run 02_osurhythm_estimator.ipynb
  3. run 03_osurhythm_momentum_estimator.ipynb
  4. do the timing of your song you want to map
  5. run 05_newsong_importer.ipynb (put the path to the new .osu in the code)
  6. run 06_osurhythm_evaluator.ipynb
  7. run 07_osuflow_evaluator_from_rhythm.ipynb
  8. if it runs successfully, it will generate a .osu file under the ipynb folder. try it out!


Hope it works!
https://discord.gg/6t897aA


Any suggestion / bug report / comment is welcome!
powaful3000
Okay, this is epic
szkiller
Noice now I can play my favorite songs that are not in osu

(I don't even know how to use this lol the instructiona are too complicated for me)
Yumeno Himiko
考虑一下开个群或者discord讨论这个东西?我也有兴趣来搞一搞

太nb啦
uv937
Looks actually pretty promising, I had this idea once
Mafumafu
Interested in seeing results from various datasets
pw384
i used to love lstm
DarkProjector
牛逼,群我垃圾
oSumAtrIX
Someone has to train it with all ranked osu maps xD
Topic Starter
[CSGA]Ar3sgice

szkiller wrote:

Noice now I can play my favorite songs that are not in osu

(I don't even know how to use this lol the instructiona are too complicated for me)
yes, that is a problem! so I'm trying to learn tensorflow.js and create a web page UI, so everyone could use it!

pw384 wrote:

i used to love lstm
在计算复杂度理论内,有限错误量子多项式时间(英语:bounded error quantum polynomial time,BQP)是一个决定性问题的复杂度类,并且其内的问题可以在多项式时间内以量子电脑解决,错误的机率小于1/3。BQP也可以视为是复杂度类BPP的量子电脑版。

Yumeno Himiko wrote:

考虑一下开个群或者discord讨论这个东西?我也有兴趣来搞一搞

太nb啦
开了个discord https://discord.gg/6t897aA
emmmmmm
mdevelopment
Woah! Pretty promising lil' project! Good job! :D
pw384

[CSGA]Ar3sgice wrote:

在计算复杂度理论内,有限错误量子多项式时间(英语:bounded error quantum polynomial time,BQP)是一个决定性问题的复杂度类,并且其内的问题可以在多项式时间内以量子电脑解决,错误的机率小于1/3。BQP也可以视为是复杂度类BPP的量子电脑版。
AR3什么时候证明BPP=P并且BQP>PH啊(
beastmode
Can't wait to see 3038 mapping!
8369048
Maybe it's easier to use deep learning in modes like taiko and mania?

等等为什么要说英语
Topic Starter
[CSGA]Ar3sgice

8369048 wrote:

Maybe it's easier to use deep learning in modes like taiko and mania?

等等为什么要说英语
It should be possible, use a classifier to estimate rhythm / synchronous note count, and let the GAN decide note color / column.

problem is I cannot play taiko and mania and therefore I don't know if a map is good or not.

理论上说是这样的,然而我……不会打taiko和mania……
oSumAtrIX
TypeError Traceback (most recent call last)
<ipython-input-3-0e701476cb0b> in <module>()
107 train_file_list = read_npz_list();
108
--> 109 train_shape, div_shape, label_shape = get_data_shape();

TypeError: 'NoneType' object is not iterable
Topic Starter
[CSGA]Ar3sgice

oSumAtrIX wrote:

TypeError Traceback (most recent call last)
<ipython-input-3-0e701476cb0b> in <module>()
107 train_file_list = read_npz_list();
108
--> 109 train_shape, div_shape, label_shape = get_data_shape();

TypeError: 'NoneType' object is not iterable
Thank you for modding! fixed (probably)
caleb5
I believe you have not set a permanent discord invite, you could send me your discord name and tag for I would like to speak with you.
Please sign in to reply.

New reply