forum

Yet another beatmap mirror + data API.

posted
Total Posts
73
Topic Starter
Hexide
Hello, I have decided to open up my mirror to public. Here you will find details how to use it and information about it.

General info:
  1. Mirror is updated every 6 hours
  2. Mirror contains only ranked beatmaps
  3. Server can upload at maximum 300mbps
How to access mirror and beatmaps:
  1. Direct link to files on the web: https://osu.hexide.com/list/
  2. Desktop application: https://mega.nz/#!pnpyTJjQ!yVtUeoGMbwz4HjJOqqb3lrfw1Me9zHQx68ZRlODC8y0 (-Source code-)
API LOCATION:
https://osu.hexide.com/

Statistical data:
SPOILER
Global network utilisation (maximum 300/300mbps):
Soon
Kyujuroku 96
Added to the osu! Projects Compilation thread. Keep the good work!!!
Topic Starter
Hexide
Started updating api to extend functionality. http://api.osu.miz.hexide.com/
Meantime search page will not be working.
Topic Starter
Hexide
Started working on small c# for multiple simultaneous beatmap downloads. It's still in extremely early stages https://github.com/Mizore/osu-mirror
Implemented few metadata lookups into api http://api.osu.miz.hexide.com/#/get/metadata/id ; http://api.osu.miz.hexide.com/#/get/metadata/id/raw

Search calls are coming soon.
Primula_old
This is really nice, thank you.
Will implement it into osu!np soon~
Topic Starter
Hexide

Primula wrote:

This is really nice, thank you.
Will implement it into osu!np soon~
That would be great! If you will have any requests/recommendations, feel free to poke me.
Topic Starter
Hexide
Coded search api. There might still be some future updated, but that wont be soon. Currently it works pretty well.
http://api.osu.miz.hexide.com/#/get/search documentations are still being written.

Also, please don't kill server. :D
power007
Very cool ftp, fast speed. I'm going to download everything xD
Topic Starter
Hexide

power007 wrote:

Very cool ftp, fast speed. I'm going to download everything xD
Cheers. :)
Topic Starter
Hexide
Also did some big updates to how search works now.
In a nutshell, you need to pass one or more search parameters. Search query will look like "/search/[param]/[param]/[param]".

Each parameter is made of 4 peaces: "[table].[column].[operand].[value]" some examples of this would be:
  1. metadata.name.like.fade
  2. maps.size.gt.100000000
  3. maps.title.like.dance
Possible operands are:
  1. eq - equal
  2. lt - less than
  3. lteq - less than or equal
  4. gt - greater than
  5. gteq - greater than or equal
  6. like - like
There are also some limitations like:
  1. You cannot use lt, lteq, gt, gteq with VARCHAR columns
  2. You cannot use like with INT and DOUBLE columns
  3. You can only use eq with ENUM columns
For list of all columns, please refer to: http://api.osu.miz.hexide.com/#/db/maps and http://api.osu.miz.hexide.com/#/db/metadata

Hopefully this will be useful for someone. :)
Topic Starter
Hexide
If any graphic artists would be willing to help me out, it would be amazing. I need someone to change some colors in few bitmaps.
Topic Starter
Hexide
Topic Starter
Hexide
Allot of progress on c# app for beatmap downloads.
Here are some images:
http://miz.hexide.com/XSgE.png
http://miz.hexide.com/bUdy.png

More coming soon! :)

If someone is interested in testing it out:

Main EXE: http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
Debug symbols: http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

For paranoid: https://github.com/Mizore/osu-mirror
nanashiRei
I like the for paranoid part, going to start doing some icons for you today. :)
Topic Starter
Hexide
Topic Starter
Hexide
past few day change digest:

Fixed registry issue where data would be be saved to incorrect location.
Added support for enter clicks on text boxes to perform search instead of clicking 'Search' button.
Files no longer will contain first capital letter in their extension.
Files will no longer be taking up more space than they should Issue #1
Added sorting support for standard list-view.
Added MP3 file download support.
Download audio files will no longer have numbers at the start of their names.
Added support for changing count of threads used for downloading file ( currently only changeable in registry )
Updated files:
Main EXE: http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
Debug symbols: http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

Also, GitHub page if you want to track progress: https://github.com/Mizore/osu-mirror
Topic Starter
Hexide
Bringing mirror down for some core improvements, could take few hours.
Back

Some more updates: Should be done within few hours.
Topic Starter
Hexide
Looks like everything is updated and restored :D , there are some major changes:
  1. Now ALL beatmap related images and other files will be under /content rather than /preview
  2. API is no longer user friendly, what i mean by that, calling /download on some beatmap will not start download with original beatmap name
  3. Removed /preview/mp3/full/download
  4. Removed /preview/image/tiny,small,medium,large (only full and custom options are left)
And as always, most recent file:
Main EXE: http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
Debug symbols: http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

https://github.com/Mizore/osu-mirror
TheAussie
the exe doesnt work just crashes...
Piotrekol

Takeda Shingen wrote:

the exe doesnt work just crashes...
and I thought I'm the only one...
this happens after clicking search
Topic Starter
Hexide
Hmm... Made small change in code as potential fix, how ever not sure if it will help.

Updated files:
Main EXE: http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
Debug symbols: http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

P.S. There will be no search results if you will not enter at least one letter into any search field.
Piotrekol
working now.
(and found duplicate entry- http://puu.sh/3v6qW.png)
Topic Starter
Hexide
Thanks for reporting, I could find actual bug in code. But what I found was that search was performed during beatmap update, since duplicate records are not in database. It should not contain duplicate result for any other similar searches, except for that exact one, since its cached.
It will clear up eventually :)

1. Yup, found few other duplicate beatmaps too. Will be fixed in few moments.
2. Fixed, those duplicate entries from search will go away once cache entry expires ( could take a while :P )
Piotrekol
My friend can't run this program under 32Bit system(Win XP)
Topic Starter
Hexide
Its .NET 4.5 app. It does not seem that it was released for WinXP.
So the only suggestion I could give now, is to upgrade.
Topic Starter
Hexide
Added option to view beatmap details on this site:


If anyone has suggestions, ideas; feel free to share. If possible I will implement them.
Topic Starter
Hexide
Added menu bar.
Moved settings to separate window.
Added ability to change concurrent download limit.
Added a bunch of links to 'Help' menu.
Recommended concurrent downloads are recommended to be kept bellow 6. Exception would be for people who have extremely good connection to beatmap mirror ( 8MB/s and up ).
Mine is set at 30 because I have direct 1Gbit connection to server.

And as always, updated files at:
http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

http://miz.hexide.com/dDld.png
http://miz.hexide.com/AbD9.png
Topic Starter
Hexide
Added option to disable question window when closing app.
Added download finished animations to download queue.
Some minor tweaks that I forgot.
Debug build: ( provides useful information in case of crashes )
http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb
Release: ( does not display debug console, slightly higher performance )
http://miz.hexide.com/mirror/Beatmap%20 ... elease.exe
Topic Starter
Hexide
Spent good while configuring nginx and added some more feature to main api: ;)
  1. Raw access to beatmap file contents: /beatmaps/12191/content/raw /beatmaps/12191/content/raw/background.jpg
  2. File hash calculations for raw beatmap files: /beatmaps/12191/content/raw/hash/noise/1.png
  3. Raw file layout: /beatmaps/12191/content/raw/layout
  4. Now all invalid requests should receive API friendly 400 response.
  5. All responses are now cached for 24 hours.
Notes:
  1. If requested beatmap was not extracted prior to accessing raw files, they will be extracted once and "Location" header will be served to request.
  2. Documentations coming soon.
Topic Starter
Hexide
It seems that there are allot of users that don't have user agent set. Who is writing apps/scripts/services that use this api, please specify user agent ( something relevant ) and if its public projects that has thread on this forum, please specify thread in Referrer header like "/t/137156".

Preferred minimal http request structure ( this is example ):
GET /search/maps.title.like.night HTTP/1.1
Host: api.osu.miz.hexide.com
User-Agent: Osu!Mirror
Referer: /t/137156
Connection: Keep-Alive
Topic Starter
Hexide
Added some basic stats: http://api.osu.miz.hexide.com/stats
ALLOT MORE coming soon.
Topic Starter
Hexide
And I broke everything. Api responses no longer contain "status" => "....".
Response code 200 = everything is good. 300 >= error.

Documentations will be updated in a moment.
show more
Please sign in to reply.

New reply