With a friend we tried doing exactly that, fetching big amount of data from osu! servers using their APIs
We searched far and wide the internet and some sneaky ways of first finding all beatmaps IDs and get them in batch somehow
We ended up empty handed with some useful tricks here and there at most
What we finally did was scraping all IDs using the API v1 with 20 ish API keys gathered form some friends for weeks
We have a copy somewhere of all beatmaps as of 2023 or something but that's far from ideal
We thought about how we would go from there, knowing what IDs to fetch to keep an up to date mirror basically
Fast forward to today, life getting in the way very little progress has been made
Because of the few ways osu! let us access their databases
A few ideas we have to suggest:
- better API or at least offer commercial uses via "big data oriented" endpoints such as a tarball of all .osu file from a range of beatmap IDs or something you get the idea
- Webhooks for the love of gods, it would benefit all of us, lowering the load on their API servers and helping developers have up to date data on a beatmap / player / category of beatmap / a specific rank / ...
In the current state of things, osu! related projets that require mass data from them is far from easy and a pain to implement
Would you mind sharing with me if you are interested in any of that other than to "poke around"? I would guess to perform some data analysis on beatmaps and players, statistics, data charts and such?