forum

altOsu new way to download beatmaps

posted
Total Posts
60
Topic Starter
Ikaer
Yo,

It's my pleasure to present you a new website: http://www.altosu.org.

It started with a simple need, every time I download a beatmap set, I have to delete every mode and every difficulty I'm not interested in (and until peppy allows us to delete beatmaps in mass when they are grouped in type or difficulties ... ;) ). I thought that if I have this need, I'm maybe not the only one, so I've created a website where you get only what you want (and to be honest, I wanna to try some tech stuff).

Things you can do for example:
- Use search box to filter on hatsune miku, use top buttons to get only the insane difficulty and the Osu! mode, sort by play count or user positive rating and then use button "Download all" to get all beatmaps of the page in a zip file.
- You like a beatmap creator ? use the search box to find it and download all its beatmaps.
- You can browse the website in beatmap mode or beatmapset mode, the beatmapset mode is just like the official website, but you can click on a difficulty icon to get only this one. (will returns soon)
- When you filter mode / difficulty / status, the download link at right of the table only get what you asked for.

This is an early state, more like an alpha, or a proof of concept to know if people are interested. And I know i'm sucking at design (hoping that a friend will help me to do something less academical, I'm thinking of a card view instead this horrible table, and header and footer and so on). Thanks to semantic UI (a css framework). I was able to do something almost easy on eyes.
What I would like to implement next:
- Pager ... and "download all" button working with more than current displayed maps.
- More filters ! Choose a specific difficulty rating for example (5 stars, 6 stars, 30 stars, etc.), chose approach rate, etc.
- Complete the search box with more entries: genre, language, tags, etc.
- User management to remember your filters.
- Add information about rankings on beatmaps.
- What you would like to see happen.



Technical note
This is also a personal project I've used to learn some technologies I'm interest in, like nodejs, mongodb, angular, etc. so be nice, it's my first rodeo with all of this.
Sources are available here: https://github.com/Ikaer/osu-listing and https://github.com/Ikaer/osu-aggregator for anyone you want to take a look behind the scene.
To simplifiy the process of how I get the beatmaps and stuff, I have background workers which crawl
- https://osu.ppy.sh/api (to get beatmap basic information)
- https://b.ppy.sh (to get images and mp3)
- official website pages (to get play count, positive user rating, and stuff like that).
And bring them together into altosu.org.
I know there is already a few songs grabbers on this forum, but you can try this one as well (with a few modification to not get mp3, image, stats, etc.) to get the osz files from bloodcat if you want.

Changelog
  1. new design, some features will return soon (sorting and display by beatmap / beatmap set)
  2. Filters are now in menu (and status have been added), sidenav have been removed for now. Loading spinner added.
  3. Sorting is available again. Add default image when missing. Add extra information on beatmap image hover.
  4. 2015/24/07 tune main query (listing of beatmaps).
  5. 2015/28/07
    1. links on beatmap cards (title, artist, creator, source, genre & language can be used to create filter now.
    2. add source and tags in searchbox
    3. fix filter with url characters like &
  6. 2015/03/08
    1. add next page / previous page
  7. 2015/15/08
    1. User management.
    2. Add song length on beatmap image hover.
    3. fix some special character in search.
  8. 2015/16/08
    1. Add a extra filters like song length or show / hide played beatmaps.
    2. Some change in menus (removed icons, sort button is now at bottom, etc.) in order to prepare a mobile version.
  9. 2015/17/08
    1. A new filter: File extensions to include in beatmap download. This is experimental, but you can if you want remove avi, wav, etc. from the download.
    2. Filters like duration, how to display played beatmaps and what type of file to download can be saved in your profile. You need to be logged in to access to the save buttons.
  10. 2015/19/09
    1. New filters are available:
      1. Difficulty (based on number and not the categories Easy, Normal, Hard, Insane, Expert, cf. https://osu.ppy.sh/wiki/Ranking_Criteria)
      2. Play duration which is different from song duration (first note to last note without breaks).
      3. Playcount
      4. Played with success count
      5. Favourited count
      6. Negative user rating
      7. Positive user rating
      8. Approved date
      9. Last update date
      10. Submitted date
H y y e i
Wow, the website is so cool. Nice!
Topic Starter
Ikaer
Thank you for the encouragement :)

Next step I think is to register user account and browse osu song directory to not propose beatmaps alreadu on disk, It pissed me the last time I used my site to download beatmaps ;)
I would also like to add rank achieved by user on beatmap cards, but historical and top ranks access are limited to 1 or 2 pages here, I don't know where to have this kind of information.
Howl
This is some hell of a good project! Here's my kudos. Keep it going well ~
Topic Starter
Ikaer
@Howl: Yes it is, but i'm inspired for now. Thanks for your kudos ! I just hope that people will give it a try :).
When I'm working on a feature, I'm trying to think as an Osu player (not too hard this far :p), hoping it will make the difference :). Or in general, as a web user, for example, list took 3 secondes to display, it was just a pain in this ass to wait --> tune the query.
About that:
update!
Optimize main query (the listing of beatmap) from 3 seconds to 300 ms, should be like instant now.
Bara-
Nice
One problem
It never shows anything when searching for maps in Help/Pending/Graveyard
Only when searching for ranked/approved
Topic Starter
Ikaer
yes, I've noticed that, the only "extra" filter that works is "qualified" (WIP works, but always returns the same two beatmaps).

The problem is that osu API (https://osu.ppy.sh/api/) does not return them or I didn't find the trick.

What I can do is parsing graveyard listing (https://osu.ppy.sh/p/beatmaplist?m=-1&r=5&g=0&la=0&ra=) for example and trying to get beatmap somewhere (not in bloodcat, I've got this message "* File not found or inaccessible!" when I'm trying), but may be by login on official website in my crawlers.

As osu player, I'm not using those categories, are they necessary ?
Bara-
For mappers and modders they are
Topic Starter
Ikaer
ok I will try to find something.
Topic Starter
Ikaer
Ok I've updated the crawlers, WIP and graveyard categories are being filled at this time.
It might take a while to get the all data of the 125 pages, after that, I think I will only do the first 10th pages for updates.

Sort is automatically updated to "last update" when WIP or / and graveyard are selected because there is no ranking date.
Topic Starter
Ikaer
User management is almost ready, but I'm not sure what Is important to show.
I think that remove / show beatmaps already downloaded is important, in order to do that, I have two possibilities;
- Create an applicatoin which browse local user folder songs
- Using osu api to get last played / scored songs

The first way is more accurate, but more intrusive, and the second way is less accurate (only the last 50 entries) bu does not require any kind of software to download.

What do you tihink ? what would you like to see after you entering your Osu! user id on the website ?
Topic Starter
Ikaer
  1. links on beatmap cards (title, artist, creator, source, genre & language) can be used to create filter now.
  2. add source and tags in searchbox
  3. fix filter with url characters like &
otoed1
Oh snap this is kool.
Guy-kun
Hey, this is looking nice.
I'd be interested to possibly consider it for the download source for T-Aiko! (https://play.google.com/store/apps/details?id=guy.taiko) in the future if you kept development up.
A few critical things are still missing like paging and a few things still unintuitive (like filtering by one game mode requires 3 clicks to disable others) but it's promising.

(P.S. Downloads on Android also do not work due to some configuration issue, osu! site has this same problem but bloodcat doesn't, I think filenames have to be non-utf8?)

Would you consider a mobile UI?
Kibbleru
As for the user downloads or w/e,
I think you shiuld find a way to implement both.

For people trying it out at first, you may not want to download anything.
If the user likes it, then they can download the full thing
Tari
It'll be cool if you could put a little bit more information in the beatmap description in each song tab.



From the above image as an example, Source should have a capital S and should have a distinctive change in font style to make it more noticeable when someone wants to look at source for the map. Also , you could put the word "Genre" on the left side to indicate that "Anime : Japanese means genre (even though its already obvious).

Also, I find it strange you labeled Mode, Difficulty, and Status on each of the Tabs and showed what they mean (like a legend) but somehow missed on or forgot to label the three buttons in the bottom of each of the song tab (http://puu.sh/jfG7f.png). In my honest opinion, you should label them :p
Topic Starter
Ikaer

Guy-kun wrote:

Hey, this is looking nice.
I'd be interested to possibly consider it for the download source for T-Aiko! (https://play.google.com/store/apps/details?id=guy.taiko) in the future if you kept development up.
A few critical things are still missing like paging and a few things still unintuitive (like filtering by one game mode requires 3 clicks to disable others) but it's promising.

(P.S. Downloads on Android also do not work due to some configuration issue, osu! site has this same problem but bloodcat doesn't, I think filenames have to be non-utf8?)

Would you consider a mobile UI?

Hi, thanks :)

For now , the mobile version is not yet started. But yes, I think to make one, especially as the card size lends itself well .

I fixed the problem on downloading beatmaps on Android (missing some HTTP headers in downloading response, including as you thought, the content-dispostion which set the file name) . Tested and approved by your app :) (nice by the way: D)

For filters you're right, I will implement a new system ( first click selects a mode, the following clicks add modes , and start again once at 4).
Topic Starter
Ikaer

Tari wrote:

It'll be cool if you could put a little bit more information in the beatmap description in each song tab.



From the above image as an example, Source should have a capital S and should have a distinctive change in font style to make it more noticeable when someone wants to look at source for the map. Also , you could put the word "Genre" on the left side to indicate that "Anime : Japanese means genre (even though its already obvious).

Also, I find it strange you labeled Mode, Difficulty, and Status on each of the Tabs and showed what they mean (like a legend) but somehow missed on or forgot to label the three buttons in the bottom of each of the song tab (http://puu.sh/jfG7f.png). In my honest opinion, you should label them :p
I will make some modifications in order to make each part more clear.
By label, you mean toolip or a popup like this: http://semantic-ui.com/modules/popup.html ?
Tari
No, I mean http://puu.sh/jfHu5.png those labels on the top :>
Topic Starter
Ikaer
@Guy-kun: I changed the behavior of difficulty and mode menus, hoping that they are more efficient.

@Tari: I added the labels to source, genre and language and use a different color for clickable links.
For the buttons "Play", "Download ", "Link", I can not put a label without losing a lot of space , cards are already big enough :) , but when mouse hovers a card , I added buttons borders, and hover them a pink border and then a tooltip after 1/2 second to explain functionality.
For additional information on beatmap (like tags, creation date, etc.) , I thought making the card clickable and then open a window with extra-information or something like that.


Probably, the next thing I will start is crawling user information when they are registered on website to add personal rank on card and played songs (/api/get_scores, /api/get_user_best and /api/get_user_recent from https://osu.ppy.sh/api/ should give pretty decent information to start with)
-Halycon
A suggestion that will greatly improve this: Perhaps an integration with the osu! song folder a player has. So it will filter out the maps they have and haven't. Players with more than 11,000 beatmaps (like I do) will very much benefit from this. Since we don't want to download duplicates. But I do like the idea, since I'm a ctb player and wanted to just see CTB and osu! std maps.
Topic Starter
Ikaer

-Halycon wrote:

A suggestion that will greatly improve this: Perhaps an integration with the osu! song folder a player has. So it will filter out the maps they have and haven't. Players with more than 11,000 beatmaps (like I do) will very much benefit from this. Since we don't want to download duplicates. But I do like the idea, since I'm a ctb player and wanted to just see CTB and osu! std maps.
Yeah that exactly the point of the website.

To manage player song folder, like I said before, it would require a third party to know that information (because browser cannot acces to anything on your computer and that's cool for security). What I can do is crawled osu api to get player last 50 plays, but that's all, and in your case with 11, 000 beatmaps, I don't think that I will have the whole thing :).

P.S I know that for some songs of the website, the last ones, generally, download does not work, I work on this.
abalabahaha
Tip: add a DNS record (A or CNAME) for the root domain, altosu.org. Currently, only http://www.altosu.org works, http://altosu.org doesn't go anywhere. Not everybody is used to typing the www. in front of domains.
Topic Starter
Ikaer
Oups, I didn't noticed that. Thanks for the tip @abalabahaha, should be ok now.
Topic Starter
Ikaer
Empty download link shouldn't happen anymore. Let me know if you have one.
Topic Starter
Ikaer

Guy-kun wrote:

A few critical things are still missing like paging
Next page and previous page are now available.
Howl
If you want to make a new post but you've already made one in the same thread recently, editing the previous post adding what you've got to say is a much preferred way than doubleposting, or in this case, tripleposting. Of course, this doesn't apply if you need to do a bump (in this case, the last reply in the thread was 8 hours ago, therefore the thread was already on top of the Development forum thread listing, and a bump wasn't required)
Topic Starter
Ikaer
@Howl: Sorry about that. Next time, I will only update my original post for minor updates, but next one is a big one :)

Update 2015/15/08
This update is centered around the user management. You can create an account and attach your Osu! id (you can find it in url of you profile account).

Account creation requires email validation to prevent robots (the sender will be altosu.org@gmail.com).

When your account is created, the server will start to recover your data (last plays and high scores) from the Osu! api.
The api limits such requests to 50 entries. So at first, there will be little data. But gradually, as you play, the history will be updated and aggregated into my database. But you can try it easily by playing a song after creating your account, and then wait a few minutes, listing should be updated with the song you've just played.


When connected, you can
- Change and save your initial modes and difficulties (in user icon button -> profile). These filters are applied each time you refresh the page, but you can always apply other filters after.
- On beatmap listing, if you have played at least one difficulty of a song, the card will appear with a blue background and a rank should be associated with the difficulty you played.
- You can choose to only display beatmaps you have played or hide them from listing.
- The letter F can appear in place of the ranking, it is because you have played this difficulty but not yet completed it.

I'm not a fan of a blue background and Osu! rank standard icons, but well, it did the job.

Misc:
- added the possibilty to filter on song length. To do that, use filter icon to deploy top menu and enter values in seconds to filter beatmaps.
- Sort button have been moved to bottom and menu buttons have a little changed to prepare the small device version.

if something goes wrong, let me know!

P.S: As a new feature, it is possible (sure) that it will have bugs. If any of these bugs require modification / new structure of database schemas, I might be forced to do a cleaning. But I hoped that it will not happen.
abraker
Oh wow, can search by difficulty. That functionality is broken here.
Topic Starter
Ikaer
2015/17/08 Update
- A new filter: File extensions to include in beatmap download. This is experimental, but you can if you want remove avi, wav, etc. from the download.
- Filters like duration, how to display played beatmaps and what type of file to download can be saved in your profile. You need to be logged in to access to the save buttons.

I think I forgot to mention it, but those extra-filters (duration, how to display played beatmaps, extension files to download) are accessible by clicking the filter icon.



dionzz99
some ranked beatmaps aren't showed. any ranked maps before 20 june 2014 aren't listed.

another issue : difficulty filter seems messed up, some maps aren't showed (excluding the maps that aren't showed in the first issue up there)

Topic Starter
Ikaer
Hi dionzz99,

For the ranked beatmaps which are not listed, I could use an example (a link to official website), it would be more easy for me to find out why they are not here.

For the difficulty, I think habits of official website have tricked you, when you filter on difficulties, let's say, Expert, list only returns beatmaps which have at least an expert mode, and when you download the beatmap, you only have this difficulty. It's like the Mode filter, when you had filter on Osu!Mania you only get this mode.

For example, if you take beatmap A, beatmap B, beatmap C (in that order) and only beatmap B has a difficulty expert, list will return beatmap B, if you add Hard difficulty and beatmap A has an hard mode, list will return beatmap A and beatmap B.

If you select all difficulties / modes, list should be exactly as official website (more or less one or two song, between 2 updates of the database).

Not sure I'm very clear :?
Rori Vidi Veni
This informative post will be deleted anyway, just move along
Topic Starter
Ikaer
I do not know anything about this game mode. Can you explain me what is a "keycount selector" ? or a link to a topic that explains.
dionzz99
Oh my god...
im so sorry for this. I was just being stupid not seeing the "Next Page" button.

once again sorry ;_;
Rirakusu
Omg i love the site. Design + Functionality > Bloodcat + official site

However some problems with downloading maps. The link you provided to download a map (be it any map) does not show it's full size, causing problems with download managers and even Chrome's built in downloader. For an example, Warak - REANIMATE (https://osu.ppy.sh/s/186318)'s full size is 7.61MB, when I downloaded from altosu.org, Chrome brings me to 8MB and stops there without completing, 3rd party download managers (any that you can imagine) might even continue it all the way to 20+MB without stopping (I use Xunlei).

Why not direct links ;;
Topic Starter
Ikaer
@jinhang_ang: Hi, first thanks :)

Then, for the downloading issues, I'm afraid, that I cannot do direct links. It would required that I prepare every possibilities of download on website server, which are countless :) (more or less difficutlies, mode, type of files, etc.).
When you click a download link, I unzip original osz files server side to get what you asked for and then I prepare a new archive (which might explain size difference with official website, compression is maybe not the same).
It's only at this point I can send in server response the real size of file, but it should be correct, and browser should know when to stop.
But It also can be an error that happened during the transfer.

I've not reproducing your issue with chrome built-in downloader and I don't know Xunlei :/. Did you apply filter on mode / difficulties / type of files before trying to download ? Can you post me here the wrong link (by using right click -> copy adress link on chrome) ? And if you are familiar with developer tools like Fiddler, you can use it to get the download response "Content-length" header and we will be able to compare yours and mine to see if something is different.
Poku
Looks very nice and useful,The design is great, I love the way you built the website, good job :)
Rori Vidi Veni
This informative post will be deleted anyway, just move along
Topic Starter
Ikaer

Rori Vidi Veni wrote:

Ikaer wrote:

I do not know anything about this game mode. Can you explain me what is a "keycount selector" ? or a link to a topic that explains.
"Display only charts with X keys" option
keycount for mania is stored in the .osu file under CircleSize
Done, you can find the filter "Circle size / key count" in top filter menu next to duration.
I've also add HP drain rate, Overall difficulty and Approach rate filter.
Like usual, if you are logged in, you can save those filters to apply them each time.

Douae
I really like this website, good job
Rori Vidi Veni
This informative post will be deleted anyway, just move along
Topic Starter
Ikaer
@PandaOG, Rori Vidi Venin, Douae; Thanks for your credits.

If you think about something else that would be usefull, don't hesitate, It will maybe not possible to do it, but if I can I will.
Topic Starter
Ikaer
Hi,

New filters are available:
- Difficulty (based on number and not the categories Easy, Normal, Hard, Insane, Expert, cf. https://osu.ppy.sh/wiki/Ranking_Criteria)
- Play duration which is different from song duration (first note to last note without breaks).
- Playcount
- Played with success count
- Favourited count
- Negative user rating
- Positive user rating
- Approved date
- Last update date
- Submitted date

You can do nice combo with dates filter for example: Approved date > 2015 and sort by plays, it removes maps which are in top 20 for like forever (cf. https://osu.ppy.sh/p/beatmaplist?l=1&r= ... &o=&r=0&q=)

All filters can be saved if you're logged in as usual.
Sviundt
Is it just me who have difficulties with logging into the site?
Topic Starter
Ikaer
It's working fine for me.
Have you never be able to login or it is just now ? There is a reset password procedure if needed.
Topic Starter
Ikaer
Filters & stuff are now in URL.
So now:
- so you can share them, for example, list of 2015 Osu!Mania beatmaps sorted by play count with only insane / expert difficulty and a key count of 7
- browser back / forward buttons are working
- F5 keep filters.
- to reset all filters, click on top Osu! icon.

+ minor fix: mp3 music sample now stop when you hit the button during the play.
Topic Starter
Ikaer
Hi,

Last feature: a rss button (at top right corner).
Each feed is unique and based on current applied filter, so you can add multiple feed in your reader.
Feed title is a direct link to the download.
Feed description contains versions embedded in download.

Tested with feedly.

I think it's the last feature I will implement except if you want something else.
Arzenvald
@.@ nice, but i sorta confused with the way mapset sorted as 'ranked, pending, qualified, approved' at once..
i didn't notice those options are select & deselect able..

maybe you can add checklist symbol (or something else like color highlight) to mark whether its on selected or deselected, its barely noticable to see the difference between transparent icons (when i select it)..

>u<)b good job, neat layout too
zenithlight
Is it possible to search based on BPM?
show more
Please sign in to reply.

New reply