forum

Auto-downloader for Bloodcat (osu! beatmap mirror)

posted
Total Posts
60
Topic Starter
RavenMac
I have been working on a small script in Python for some time now and I think some people would appreciate.
I have always, from when I first started playing osu, wanted to be up-to-date with all of the new maps that are available.
Keeping up with the beatmap packs is a bit of a hassle, and somehow people are still having issues with them.

What I've made is a auto-downloader that works through Bloodcat (my favorite beatmap mirror).
Just a quick explanation as to what it does.
SPOILER
Upon starting the script, it asks where you would like to download the beatmaps to.
  • This could be your downloads file, songs folder, anywhere on your computer.
It then asks if you want to save the filename using unicode characters.
  • It will continue saving files this way until you close and reopen the script.
    I just decided to add this option. Not really useful, but gives you an option.
    Notice, not all songs use unicode characters.
It checks the JSON info from bloodcat and compares it to the Recent.json file.
If you used my program before this update, it will compare the JSON info with the Recent.num file.
Once finished with the check and/or downloads, it will delete the Recent.num file, and save using the Recent.json file.
  • The Recent.json file contains the JSON data for the songs.
    If you just updated to the newest version, it will check with the older Recent.num file, delete this and remake it as a Recent.json file instead.
    If Recent.json doesn't exist, it will be created after the first download is complete, continue to update after every download, and update once more after all downloads are finished.
If there are any new files from the JSON info, then they will be downloaded one at a time to the destination previously selected.
  • Information about the song is displayed on the python command line as the download starts.
    This will not cause the command line to pop-up. It's just to display more info if you just so happen to be looking at it.
    Again, if Recent.json doesn't exists, it will download everything available, or if you have the older Recent.num file, it will check that first and replace the file with the newer Recent.json file.
    A lot of these files seem to be updates to other songs already available.
After each download, it will update the Recent.num file with what was just downloaded.
After all downloads are finished, it will then update Recent.json file again.
It will wait 5 minutes, then check the JSON data again in an endless loop until the command window is closed.
There is a 5 second timeout for checking the JSON data and a 5 minute timeout for downloads. If a timeout were to occur, the program will beep, wait 5 seconds and attempt to connect again.
If there are 5 failed attempts or more in a row, it will wait 5 minutes before trying again.
  • If connection errors still persist, the only thing I could think the problem would be that you cannot connect to bloodcat.com/osu/
    If you can connect to bloodcat and still get errors, please message me the error message you are getting.

I know the code isn't pretty or the best. I'm still learning Python, but it works on my side and wanted to share.
I've also gotten hold of the owner of bloodcat. "I'm very happy that people makes application with my site." -방성환
I just wanted to get the ok from the owner before posting the code for everyone to use.

Downloads
One thing you will need before running this script is either Python 2.7 or 3.4.
The current version of both are 2.7.8 and 3.4.1.

Now for the script.
The code for Python 2.7 can be either viewed of downloaded from here.
The code for Python 3.4 can be either viewed of downloaded from here.
This only includes the auto-download.py file. The Recent.num file will be created when you finish a download.

I am trying to learn how to make a GUI for it. I will update this page if/when I ever get finished with it.
I hope you enjoy it.

P.S.: Once started, you can minimize the window. It will run in the background and beep if there are any errors.

P.P.S.: As of Aug. 14, 2014, I've been taking my Advanced Systems Project class and started making a GUI application in Java. I will be learning more in depth about GUIs in the next few months.

Updates
SPOILER
Mar. 9, 2016
- Fixed compatibility with updated JSON format from Bloodcat.
- Changed how itemInfo is displayed due to JSON change.
- Changed Recent.json formatting due to JSON change.
- Happy Miku Day! (i didn't plan on this, but o well)

Oct. 12, 2014
- Fixed error unicode in ASCII title.
- Use enumerate(to_download) instead of range(len(to_download))
(Just because i learned something new in python :D)
- Put beep method on two lines and a space.
(makes it easier to just edit what i have online instead of re-uploading the file and change the links here)
(mediafire kept changing the first tab as four spaces. Changed it to a space and it fixed this.)

Sep. 1, 2014
- Fixed error when updating code.

Aug. 27, 2014
- Slight download display.
(Shows song info only once.)
(Shows number of songs left to download (Ex. 2 of 5))
- Changed error messages to help with possible debugging.
- Changed and added functions.

Aug. 26, 2014
- Now checks 100 items
- Saves using JSON data in case of quick updates
- Added timeouts for connections and longer waits if multiple timeouts occur
- Added comments for anyone wanting to learn
- This may be my final python update. I am thinking of continuing this with Java.

June 6, 2014
- Fixed an issue displaying song info.
(bloodcat now stores all items as strings)
- Updated location to download Python. (new versions mentioned as well)
- Status code '8' added. For Qualified songs.

Apr. 27, 2014
- Fixed an issue trying to display the 'source' key.
(bloodcat no longer displays the source for the songs, which caused this issue)

Nov. 19, 2013
- I'll add my edits to this page in here.
- Added option to 2.7 code to save file with unicode characters.
- Uploaded 3.3 code.
- Change script explanation for added update.
Hexide
Off to a good start, just one thing I noticed:
conn = urllib2.urlopen('http://bloodcat.com/osu/?mod=json')
data = conn.read().split('},{')
conn.close()
items, s = [], ['{"id":',
',"artist":"',
'","source":',
'","title":"',
'","date":']
for l in data:
items.append([l[l.find(s[0])+len(s[0]):l.find(s[1])],
l[l.find(s[1])+len(s[1]):l.find(s[2])]+' - '+
l[l.find(s[3])+len(s[3]):l.find(s[4])]])
Doing it like that maybe works, but might break with slightest layout change ( even when actual data behind it is same ).
So you are better off just using standard JSON parsing functions.
Avail_old_1
import simplejson

url = "http://bloodcat.com/osu/?mod=json"
json = simplejson.load(url)

id = json[0]['id']
artist = json[0]['artist']
source = json[0]['source']
title = json[0]['title']
date = json[0]['date']

Or something like that.
Topic Starter
RavenMac
To be honest, i only thought JSON was something for bloodcat.
I've never herd of if before looking at the information page.
There are still a lot of things with Python i still don't know about.

Thank you Hexide and Avail for showing me this.
I'll look into this a little more and will see about updating it.

Edit: I have changed the code a little bit to use the json module.
I will have this test for about an hour while i get lunch.
If there are no errors by the time i get back, i'll update my code using the same link on my first post.

Edit 2: It's been running well for an hour now and have just updated the file. Same link in the first post.
ps. i love mediafire's edit file. Don't have to reupload files if it's a small change and keeps the same link.
Topic Starter
RavenMac
I have just finished the code working with Python 3.3 and added a couple extra things.
Piotrekol
Good way to start!
(I would release my downloader months ago but it's based on some parts of osu site(but uses bloodcat to dl) so it's no-go for mass use)
suggestions:
-Ask if user want to download only ranked beatmap sets or all (or only unranked?)
-ability to download upto ~60(or more)last ranked maps(I think I could provide public api for this- not promising tough)
Topic Starter
RavenMac

Piotrekol wrote:

-Ask if user want to download only ranked beatmap sets or all (or only unranked?)
This wouldn't be too difficult to add.
Having 2+ windows pop-up isn't something i like for small options like this.
Definitely something to remember if / when i make a GUI for it.

Piotrekol wrote:

-ability to download upto ~60(or more)last ranked maps(I think I could provide public api for this- not promising tough)
I don't think checking the last 60 updated songs every time will do much. Maybe at just the start.
20 is what json brings up by default, and that seems to work quite well.
- Marco -
22:45:04 - Downloading: 60309 nano - Omoide Kakera
Creator: 1065180 Smoothie
(<class 'KeyError'>, KeyError('source',), <traceback object at 0x00000000065E1E0
8>)

And it will continue beeping :c
Woddles

marcostudios wrote:

22:45:04 - Downloading: 60309 nano - Omoide Kakera
Creator: 1065180 Smoothie
(<class 'KeyError'>, KeyError('source',), <traceback object at 0x00000000065E1E0
8>)

And it will continue beeping :c
Thread is about 5months old, would not surprise me if the program was broken by an update to the bloodcat website or something similar
Topic Starter
RavenMac

marcostudios wrote:

22:45:04 - Downloading: 60309 nano - Omoide Kakera
Creator: 1065180 Smoothie
(<class 'KeyError'>, KeyError('source',), <traceback object at 0x00000000065E1E0
8>)

And it will continue beeping :c
I'm sorry for not replying earlier.
I've fixed the issue. Bloodcat no longer provides the source of the songs any more.

I've been busy with my Java class in college and have had hardly any time for anything else.
I'm thinking of making a Java version of this. Almost everyone has java so no need to install python or downloading the appropriate version of the code depending on what version of python you're using.
Also, GUI's in Java is much easier to understand than python, imo.
I'm still not very good with them though.
Topic Starter
RavenMac
Just updated my program again. Bloodcat no longer stores the genre or status as an integer. Everything is strings.
Was asked by neptune23 how to get it working and decided to check to see if it was still working. It wasn't.

I can no longer use my own program every day. My ISP has put a data cap on my house. 300GB split 5 ways.
One more reason why i hate Comcast. No matter how fast your internet is, the cap is the same.
And I've checked other ISPs. It's all i have in the area...

Edit.
One more small edit. I don't know why this wan't in there in the first place.
Added status code 8 for Qualified maps.
Aistify
How do we make it so that it downloads to a specific folder without asking us to later on?
Topic Starter
RavenMac

Aistify wrote:

How do we make it so that it downloads to a specific folder without asking us to later on?
I did not add anything like that to this program. I haven't touched this program for quite some time now, so it's kinda dated.
321jurgen
It throws me a permission error while it downloaded the map succesfully.
http://puu.sh/9KW6g.png
Topic Starter
RavenMac

321jurgen wrote:

It throws me a permission error while it downloaded the map succesfully.
http://puu.sh/9KW6g.png
I'll take some time to look into this tonight. I'm a bit busy setting up my computer and a few other things while watching over someone's house for a few days.
YAY for no data caps over here.
Topic Starter
RavenMac
I've looked around and the only things i can find are:
1) The file is being used by something already, but the file is saved once the program has the entire file and once the file is opened by the default program (osu!) it's moved to the songs directory and unpackaged as all the files for the map. (what i guess happens. i could be wrong, but not too far off)
2) You or the user you are logged in as doesn't have permission to write to that directory. However, it looks like you were able to write one file there in the first place... I assume you're the owner of the computer and also have admin privileges. If not, try with the admin/owner of the computer.

I made a small script to check if you've got permission to write to a directory. You can paste it into a text file, make sure it has the .py extension, and the save type as "all files" to prevent it from making it a .txt file. If it passes, try the downloader again in the same folder.

And for anyone else who wants to try this, this is for python 3.x.
import tkinter, tkinter.filedialog, tkinter.messagebox, os
root = tkinter.Tk()
root.withdraw()
folder = tkinter.filedialog.askdirectory(parent=root, initialdir='/', title='Select folder to test read/write permissions.')

if os.access(folder, os.W_OK):
if os.access(folder, os.R_OK):
tkinter.messagebox.showinfo('Directory permission','You are allowed to read/write here.')
else:
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to write here.')
else:
if os.access(folder, os.R_OK):
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to read here.')
else:
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to read/write here.')
321jurgen

RavenMac wrote:

I've looked around and the only things i can find are:
1) The file is being used by something already, but the file is saved once the program has the entire file and once the file is opened by the default program (osu!) it's moved to the songs directory and unpackaged as all the files for the map. (what i guess happens. i could be wrong, but not too far off)
2) You or the user you are logged in as doesn't have permission to write to that directory. However, it looks like you were able to write one file there in the first place... I assume you're the owner of the computer and also have admin privileges. If not, try with the admin/owner of the computer.

I made a small script to check if you've got permission to write to a directory. You can paste it into a text file, make sure it has the .py extension, and the save type as "all files" to prevent it from making it a .txt file. If it passes, try the downloader again in the same folder.

And for anyone else who wants to try this, this is for python 3.x.
import tkinter, tkinter.filedialog, tkinter.messagebox, os
root = tkinter.Tk()
root.withdraw()
folder = tkinter.filedialog.askdirectory(parent=root, initialdir='/', title='Select folder to test read/write permissions.')

if os.access(folder, os.W_OK):
if os.access(folder, os.R_OK):
tkinter.messagebox.showinfo('Directory permission','You are allowed to read/write here.')
else:
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to write here.')
else:
if os.access(folder, os.R_OK):
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to read here.')
else:
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to read/write here.')
I don't know how but it seems to have fixed itself. Thanks alot though for looking into it.
Sandalot
RavenMac, you should make it in java. 1+ is a good idea
Also if it sorted all the ranked from the unranked in 2 folders and kept that up to date would be an awesome addition.
321jurgen
So this is maybe a bit far fetched but it would be awesome if you could make a program that runs in the background that downloads and updates all the maps. Maybe I should start programming something myself sometime :/ I only do web related stuff and it would be good to explore some different areas. If so in what language should I write it?
Sandalot

321jurgen wrote:

So this is maybe a bit far fetched but it would be awesome if you could make a program that runs in the background that downloads and updates all the maps. Maybe I should start programming something myself sometime :/ I only do web related stuff and it would be good to explore some different areas. If so in what language should I write it?
Python would be a good place to start but java would make it a lot more easier for other people to use it.

I had the same idea as you.
show more
Please sign in to reply.

New reply