Guest
home
beatmaps
rankings
community
store
help
forum
  1. Forums
  2. osu!
  3. Development

altOsu new way to download beatmaps

posted
Total Posts
60
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
Yo,

It's my pleasure to present you a new website: http://www.altosu.org.

It started with a simple need, every time I download a beatmap set, I have to delete every mode and every difficulty I'm not interested in (and until peppy allows us to delete beatmaps in mass when they are grouped in type or difficulties ... ;) ). I thought that if I have this need, I'm maybe not the only one, so I've created a website where you get only what you want (and to be honest, I wanna to try some tech stuff).

Things you can do for example:
- Use search box to filter on hatsune miku, use top buttons to get only the insane difficulty and the Osu! mode, sort by play count or user positive rating and then use button "Download all" to get all beatmaps of the page in a zip file.
- You like a beatmap creator ? use the search box to find it and download all its beatmaps.
- You can browse the website in beatmap mode or beatmapset mode, the beatmapset mode is just like the official website, but you can click on a difficulty icon to get only this one. (will returns soon)
- When you filter mode / difficulty / status, the download link at right of the table only get what you asked for.

This is an early state, more like an alpha, or a proof of concept to know if people are interested. And I know i'm sucking at design (hoping that a friend will help me to do something less academical, I'm thinking of a card view instead this horrible table, and header and footer and so on). Thanks to semantic UI (a css framework). I was able to do something almost easy on eyes.
What I would like to implement next:
- Pager ... and "download all" button working with more than current displayed maps.
- More filters ! Choose a specific difficulty rating for example (5 stars, 6 stars, 30 stars, etc.), chose approach rate, etc.
- Complete the search box with more entries: genre, language, tags, etc.
- User management to remember your filters.
- Add information about rankings on beatmaps.
- What you would like to see happen.



Technical note
This is also a personal project I've used to learn some technologies I'm interest in, like nodejs, mongodb, angular, etc. so be nice, it's my first rodeo with all of this.
Sources are available here: https://github.com/Ikaer/osu-listing and https://github.com/Ikaer/osu-aggregator for anyone you want to take a look behind the scene.
To simplifiy the process of how I get the beatmaps and stuff, I have background workers which crawl
- https://osu.ppy.sh/api (to get beatmap basic information)
- https://b.ppy.sh (to get images and mp3)
- official website pages (to get play count, positive user rating, and stuff like that).
And bring them together into altosu.org.
I know there is already a few songs grabbers on this forum, but you can try this one as well (with a few modification to not get mp3, image, stats, etc.) to get the osz files from bloodcat if you want.

Changelog
  1. new design, some features will return soon (sorting and display by beatmap / beatmap set)
  2. Filters are now in menu (and status have been added), sidenav have been removed for now. Loading spinner added.
  3. Sorting is available again. Add default image when missing. Add extra information on beatmap image hover.
  4. 2015/24/07 tune main query (listing of beatmaps).
  5. 2015/28/07
    1. links on beatmap cards (title, artist, creator, source, genre & language can be used to create filter now.
    2. add source and tags in searchbox
    3. fix filter with url characters like &
  6. 2015/03/08
    1. add next page / previous page
  7. 2015/15/08
    1. User management.
    2. Add song length on beatmap image hover.
    3. fix some special character in search.
  8. 2015/16/08
    1. Add a extra filters like song length or show / hide played beatmaps.
    2. Some change in menus (removed icons, sort button is now at bottom, etc.) in order to prepare a mobile version.
  9. 2015/17/08
    1. A new filter: File extensions to include in beatmap download. This is experimental, but you can if you want remove avi, wav, etc. from the download.
    2. Filters like duration, how to display played beatmaps and what type of file to download can be saved in your profile. You need to be logged in to access to the save buttons.
  10. 2015/19/09
    1. New filters are available:
      1. Difficulty (based on number and not the categories Easy, Normal, Hard, Insane, Expert, cf. https://osu.ppy.sh/wiki/Ranking_Criteria)
      2. Play duration which is different from song duration (first note to last note without breaks).
      3. Playcount
      4. Played with success count
      5. Favourited count
      6. Negative user rating
      7. Positive user rating
      8. Approved date
      9. Last update date
      10. Submitted date
Last edited by Ikaer , edited 15 times in total.
H y y e i
H y y e i
Wow, the website is so cool. Nice!
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
Thank you for the encouragement :)

Next step I think is to register user account and browse osu song directory to not propose beatmaps alreadu on disk, It pissed me the last time I used my site to download beatmaps ;)
I would also like to add rank achieved by user on beatmap cards, but historical and top ranks access are limited to 1 or 2 pages here, I don't know where to have this kind of information.
Howl
1,121 posts
Joined June 2013
Howl
This is some hell of a good project! Here's my kudos. Keep it going well ~
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
@Howl: Yes it is, but i'm inspired for now. Thanks for your kudos ! I just hope that people will give it a try :).
When I'm working on a feature, I'm trying to think as an Osu player (not too hard this far :p), hoping it will make the difference :). Or in general, as a web user, for example, list took 3 secondes to display, it was just a pain in this ass to wait --> tune the query.
About that:
update!
Optimize main query (the listing of beatmap) from 3 seconds to 300 ms, should be like instant now.
Bara-
11,955 posts
Joined April 2013
Bara-
Nice
One problem
It never shows anything when searching for maps in Help/Pending/Graveyard
Only when searching for ranked/approved
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
yes, I've noticed that, the only "extra" filter that works is "qualified" (WIP works, but always returns the same two beatmaps).

The problem is that osu API (https://osu.ppy.sh/api/) does not return them or I didn't find the trick.

What I can do is parsing graveyard listing (https://osu.ppy.sh/p/beatmaplist?m=-1&r=5&g=0&la=0&ra=) for example and trying to get beatmap somewhere (not in bloodcat, I've got this message "* File not found or inaccessible!" when I'm trying), but may be by login on official website in my crawlers.

As osu player, I'm not using those categories, are they necessary ?
Bara-
11,955 posts
Joined April 2013
Bara-
For mappers and modders they are
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
ok I will try to find something.
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
Ok I've updated the crawlers, WIP and graveyard categories are being filled at this time.
It might take a while to get the all data of the 125 pages, after that, I think I will only do the first 10th pages for updates.

Sort is automatically updated to "last update" when WIP or / and graveyard are selected because there is no ranking date.
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
User management is almost ready, but I'm not sure what Is important to show.
I think that remove / show beatmaps already downloaded is important, in order to do that, I have two possibilities;
- Create an applicatoin which browse local user folder songs
- Using osu api to get last played / scored songs

The first way is more accurate, but more intrusive, and the second way is less accurate (only the last 50 entries) bu does not require any kind of software to download.

What do you tihink ? what would you like to see after you entering your Osu! user id on the website ?
Last edited by Ikaer , edited 2 times in total.
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
  1. links on beatmap cards (title, artist, creator, source, genre & language) can be used to create filter now.
  2. add source and tags in searchbox
  3. fix filter with url characters like &
otoed1
639 posts
Joined May 2014
otoed1
Oh snap this is kool.
Guy-kun
770 posts
Joined December 2009
Guy-kun
Hey, this is looking nice.
I'd be interested to possibly consider it for the download source for T-Aiko! (https://play.google.com/store/apps/details?id=guy.taiko) in the future if you kept development up.
A few critical things are still missing like paging and a few things still unintuitive (like filtering by one game mode requires 3 clicks to disable others) but it's promising.

(P.S. Downloads on Android also do not work due to some configuration issue, osu! site has this same problem but bloodcat doesn't, I think filenames have to be non-utf8?)

Would you consider a mobile UI?
Kibbleru
osu! Alumni
5,844 posts
Joined August 2013
Kibbleru
As for the user downloads or w/e,
I think you shiuld find a way to implement both.

For people trying it out at first, you may not want to download anything.
If the user likes it, then they can download the full thing
Tari
osu! Alumni
1,671 posts
Joined July 2012
Tari
It'll be cool if you could put a little bit more information in the beatmap description in each song tab.



From the above image as an example, Source should have a capital S and should have a distinctive change in font style to make it more noticeable when someone wants to look at source for the map. Also , you could put the word "Genre" on the left side to indicate that "Anime : Japanese means genre (even though its already obvious).

Also, I find it strange you labeled Mode, Difficulty, and Status on each of the Tabs and showed what they mean (like a legend) but somehow missed on or forgot to label the three buttons in the bottom of each of the song tab (http://puu.sh/jfG7f.png). In my honest opinion, you should label them :p
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer

Guy-kun wrote:

Hey, this is looking nice.
I'd be interested to possibly consider it for the download source for T-Aiko! (https://play.google.com/store/apps/details?id=guy.taiko) in the future if you kept development up.
A few critical things are still missing like paging and a few things still unintuitive (like filtering by one game mode requires 3 clicks to disable others) but it's promising.

(P.S. Downloads on Android also do not work due to some configuration issue, osu! site has this same problem but bloodcat doesn't, I think filenames have to be non-utf8?)

Would you consider a mobile UI?

Hi, thanks :)

For now , the mobile version is not yet started. But yes, I think to make one, especially as the card size lends itself well .

I fixed the problem on downloading beatmaps on Android (missing some HTTP headers in downloading response, including as you thought, the content-dispostion which set the file name) . Tested and approved by your app :) (nice by the way: D)

For filters you're right, I will implement a new system ( first click selects a mode, the following clicks add modes , and start again once at 4).
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer

Tari wrote:

It'll be cool if you could put a little bit more information in the beatmap description in each song tab.



From the above image as an example, Source should have a capital S and should have a distinctive change in font style to make it more noticeable when someone wants to look at source for the map. Also , you could put the word "Genre" on the left side to indicate that "Anime : Japanese means genre (even though its already obvious).

Also, I find it strange you labeled Mode, Difficulty, and Status on each of the Tabs and showed what they mean (like a legend) but somehow missed on or forgot to label the three buttons in the bottom of each of the song tab (http://puu.sh/jfG7f.png). In my honest opinion, you should label them :p
I will make some modifications in order to make each part more clear.
By label, you mean toolip or a popup like this: http://semantic-ui.com/modules/popup.html ?
Tari
osu! Alumni
1,671 posts
Joined July 2012
Tari
No, I mean http://puu.sh/jfHu5.png those labels on the top :>
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
@Guy-kun: I changed the behavior of difficulty and mode menus, hoping that they are more efficient.

@Tari: I added the labels to source, genre and language and use a different color for clickable links.
For the buttons "Play", "Download ", "Link", I can not put a label without losing a lot of space , cards are already big enough :) , but when mouse hovers a card , I added buttons borders, and hover them a pink border and then a tooltip after 1/2 second to explain functionality.
For additional information on beatmap (like tags, creation date, etc.) , I thought making the card clickable and then open a window with extra-information or something like that.


Probably, the next thing I will start is crawling user information when they are registered on website to add personal rank on card and played songs (/api/get_scores, /api/get_user_best and /api/get_user_recent from https://osu.ppy.sh/api/ should give pretty decent information to start with)
-Halycon
6 posts
Joined March 2015
-Halycon
A suggestion that will greatly improve this: Perhaps an integration with the osu! song folder a player has. So it will filter out the maps they have and haven't. Players with more than 11,000 beatmaps (like I do) will very much benefit from this. Since we don't want to download duplicates. But I do like the idea, since I'm a ctb player and wanted to just see CTB and osu! std maps.
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer

-Halycon wrote:

A suggestion that will greatly improve this: Perhaps an integration with the osu! song folder a player has. So it will filter out the maps they have and haven't. Players with more than 11,000 beatmaps (like I do) will very much benefit from this. Since we don't want to download duplicates. But I do like the idea, since I'm a ctb player and wanted to just see CTB and osu! std maps.
Yeah that exactly the point of the website.

To manage player song folder, like I said before, it would require a third party to know that information (because browser cannot acces to anything on your computer and that's cool for security). What I can do is crawled osu api to get player last 50 plays, but that's all, and in your case with 11, 000 beatmaps, I don't think that I will have the whole thing :).

P.S I know that for some songs of the website, the last ones, generally, download does not work, I work on this.
abalabahaha
18 posts
Joined April 2014
abalabahaha
Tip: add a DNS record (A or CNAME) for the root domain, altosu.org. Currently, only http://www.altosu.org works, http://altosu.org doesn't go anywhere. Not everybody is used to typing the www. in front of domains.
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
Oups, I didn't noticed that. Thanks for the tip @abalabahaha, should be ok now.
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
Empty download link shouldn't happen anymore. Let me know if you have one.
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer

Guy-kun wrote:

A few critical things are still missing like paging
Next page and previous page are now available.
Howl
1,121 posts
Joined June 2013
Howl
If you want to make a new post but you've already made one in the same thread recently, editing the previous post adding what you've got to say is a much preferred way than doubleposting, or in this case, tripleposting. Of course, this doesn't apply if you need to do a bump (in this case, the last reply in the thread was 8 hours ago, therefore the thread was already on top of the Development forum thread listing, and a bump wasn't required)
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
@Howl: Sorry about that. Next time, I will only update my original post for minor updates, but next one is a big one :)

Update 2015/15/08
This update is centered around the user management. You can create an account and attach your Osu! id (you can find it in url of you profile account).

Account creation requires email validation to prevent robots (the sender will be altosu.org@gmail.com).

When your account is created, the server will start to recover your data (last plays and high scores) from the Osu! api.
The api limits such requests to 50 entries. So at first, there will be little data. But gradually, as you play, the history will be updated and aggregated into my database. But you can try it easily by playing a song after creating your account, and then wait a few minutes, listing should be updated with the song you've just played.


When connected, you can
- Change and save your initial modes and difficulties (in user icon button -> profile). These filters are applied each time you refresh the page, but you can always apply other filters after.
- On beatmap listing, if you have played at least one difficulty of a song, the card will appear with a blue background and a rank should be associated with the difficulty you played.
- You can choose to only display beatmaps you have played or hide them from listing.
- The letter F can appear in place of the ranking, it is because you have played this difficulty but not yet completed it.

I'm not a fan of a blue background and Osu! rank standard icons, but well, it did the job.

Misc:
- added the possibilty to filter on song length. To do that, use filter icon to deploy top menu and enter values in seconds to filter beatmaps.
- Sort button have been moved to bottom and menu buttons have a little changed to prepare the small device version.

if something goes wrong, let me know!

P.S: As a new feature, it is possible (sure) that it will have bugs. If any of these bugs require modification / new structure of database schemas, I might be forced to do a cleaning. But I hoped that it will not happen.
Last edited by Ikaer , edited 1 time in total.
abraker
Global Moderator
8,295 posts
Joined July 2014
abraker
Oh wow, can search by difficulty. That functionality is broken here.
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
2015/17/08 Update
- A new filter: File extensions to include in beatmap download. This is experimental, but you can if you want remove avi, wav, etc. from the download.
- Filters like duration, how to display played beatmaps and what type of file to download can be saved in your profile. You need to be logged in to access to the save buttons.

I think I forgot to mention it, but those extra-filters (duration, how to display played beatmaps, extension files to download) are accessible by clicking the filter icon.



dionzz99
677 posts
Joined April 2013
dionzz99
some ranked beatmaps aren't showed. any ranked maps before 20 june 2014 aren't listed.

another issue : difficulty filter seems messed up, some maps aren't showed (excluding the maps that aren't showed in the first issue up there)

X only
I and X
HIX
Ikaer
46 posts
Joined June 2014
Topic Starter
Ikaer
Hi dionzz99,

For the ranked beatmaps which are not listed, I could use an example (a link to official website), it would be more easy for me to find out why they are not here.

For the difficulty, I think habits of official website have tricked you, when you filter on difficulties, let's say, Expert, list only returns beatmaps which have at least an expert mode, and when you download the beatmap, you only have this difficulty. It's like the Mode filter, when you had filter on Osu!Mania you only get this mode.

For example, if you take beatmap A, beatmap B, beatmap C (in that order) and only beatmap B has a difficulty expert, list will return beatmap B, if you add Hard difficulty and beatmap A has an hard mode, list will return beatmap A and beatmap B.

If you select all difficulties / modes, list should be exactly as official website (more or less one or two song, between 2 updates of the database).

Not sure I'm very clear :?
Rori Vidi Veni
175 posts
Joined January 2013
Rori Vidi Veni
This informative post will be deleted anyway, just move along
Last edited by Rori Vidi Veni , edited 1 time in total.
show more
Please sign in to reply.

New reply

0
1 / 60