Auto-downloader for Bloodcat (osu! beatmap mirror)

posted
Total Posts
60
show more
Topic Starter
RavenMac

Piotrekol wrote:

-Ask if user want to download only ranked beatmap sets or all (or only unranked?)
This wouldn't be too difficult to add.
Having 2+ windows pop-up isn't something i like for small options like this.
Definitely something to remember if / when i make a GUI for it.

Piotrekol wrote:

-ability to download upto ~60(or more)last ranked maps(I think I could provide public api for this- not promising tough)
I don't think checking the last 60 updated songs every time will do much. Maybe at just the start.
20 is what json brings up by default, and that seems to work quite well.
- Marco -
22:45:04 - Downloading: 60309 nano - Omoide Kakera
Creator: 1065180 Smoothie
(<class 'KeyError'>, KeyError('source',), <traceback object at 0x00000000065E1E0
8>)

And it will continue beeping :c
Woddles

marcostudios wrote:

22:45:04 - Downloading: 60309 nano - Omoide Kakera
Creator: 1065180 Smoothie
(<class 'KeyError'>, KeyError('source',), <traceback object at 0x00000000065E1E0
8>)

And it will continue beeping :c
Thread is about 5months old, would not surprise me if the program was broken by an update to the bloodcat website or something similar
Topic Starter
RavenMac

marcostudios wrote:

22:45:04 - Downloading: 60309 nano - Omoide Kakera
Creator: 1065180 Smoothie
(<class 'KeyError'>, KeyError('source',), <traceback object at 0x00000000065E1E0
8>)

And it will continue beeping :c
I'm sorry for not replying earlier.
I've fixed the issue. Bloodcat no longer provides the source of the songs any more.

I've been busy with my Java class in college and have had hardly any time for anything else.
I'm thinking of making a Java version of this. Almost everyone has java so no need to install python or downloading the appropriate version of the code depending on what version of python you're using.
Also, GUI's in Java is much easier to understand than python, imo.
I'm still not very good with them though.
Topic Starter
RavenMac
Just updated my program again. Bloodcat no longer stores the genre or status as an integer. Everything is strings.
Was asked by neptune23 how to get it working and decided to check to see if it was still working. It wasn't.

I can no longer use my own program every day. My ISP has put a data cap on my house. 300GB split 5 ways.
One more reason why i hate Comcast. No matter how fast your internet is, the cap is the same.
And I've checked other ISPs. It's all i have in the area...

Edit.
One more small edit. I don't know why this wan't in there in the first place.
Added status code 8 for Qualified maps.
Aistify
How do we make it so that it downloads to a specific folder without asking us to later on?
Topic Starter
RavenMac

Aistify wrote:

How do we make it so that it downloads to a specific folder without asking us to later on?
I did not add anything like that to this program. I haven't touched this program for quite some time now, so it's kinda dated.
321jurgen
It throws me a permission error while it downloaded the map succesfully.
http://puu.sh/9KW6g.png
Topic Starter
RavenMac

321jurgen wrote:

It throws me a permission error while it downloaded the map succesfully.
http://puu.sh/9KW6g.png
I'll take some time to look into this tonight. I'm a bit busy setting up my computer and a few other things while watching over someone's house for a few days.
YAY for no data caps over here.
Topic Starter
RavenMac
I've looked around and the only things i can find are:
1) The file is being used by something already, but the file is saved once the program has the entire file and once the file is opened by the default program (osu!) it's moved to the songs directory and unpackaged as all the files for the map. (what i guess happens. i could be wrong, but not too far off)
2) You or the user you are logged in as doesn't have permission to write to that directory. However, it looks like you were able to write one file there in the first place... I assume you're the owner of the computer and also have admin privileges. If not, try with the admin/owner of the computer.

I made a small script to check if you've got permission to write to a directory. You can paste it into a text file, make sure it has the .py extension, and the save type as "all files" to prevent it from making it a .txt file. If it passes, try the downloader again in the same folder.

And for anyone else who wants to try this, this is for python 3.x.
import tkinter, tkinter.filedialog, tkinter.messagebox, os
root = tkinter.Tk()
root.withdraw()
folder = tkinter.filedialog.askdirectory(parent=root, initialdir='/', title='Select folder to test read/write permissions.')

if os.access(folder, os.W_OK):
if os.access(folder, os.R_OK):
tkinter.messagebox.showinfo('Directory permission','You are allowed to read/write here.')
else:
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to write here.')
else:
if os.access(folder, os.R_OK):
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to read here.')
else:
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to read/write here.')
321jurgen

RavenMac wrote:

I've looked around and the only things i can find are:
1) The file is being used by something already, but the file is saved once the program has the entire file and once the file is opened by the default program (osu!) it's moved to the songs directory and unpackaged as all the files for the map. (what i guess happens. i could be wrong, but not too far off)
2) You or the user you are logged in as doesn't have permission to write to that directory. However, it looks like you were able to write one file there in the first place... I assume you're the owner of the computer and also have admin privileges. If not, try with the admin/owner of the computer.

I made a small script to check if you've got permission to write to a directory. You can paste it into a text file, make sure it has the .py extension, and the save type as "all files" to prevent it from making it a .txt file. If it passes, try the downloader again in the same folder.

And for anyone else who wants to try this, this is for python 3.x.
import tkinter, tkinter.filedialog, tkinter.messagebox, os
root = tkinter.Tk()
root.withdraw()
folder = tkinter.filedialog.askdirectory(parent=root, initialdir='/', title='Select folder to test read/write permissions.')

if os.access(folder, os.W_OK):
if os.access(folder, os.R_OK):
tkinter.messagebox.showinfo('Directory permission','You are allowed to read/write here.')
else:
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to write here.')
else:
if os.access(folder, os.R_OK):
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to read here.')
else:
tkinter.messagebox.showinfo('Directory permission','You are NOT allowed to read/write here.')
I don't know how but it seems to have fixed itself. Thanks alot though for looking into it.
Sandalot
RavenMac, you should make it in java. 1+ is a good idea
Also if it sorted all the ranked from the unranked in 2 folders and kept that up to date would be an awesome addition.
321jurgen
So this is maybe a bit far fetched but it would be awesome if you could make a program that runs in the background that downloads and updates all the maps. Maybe I should start programming something myself sometime :/ I only do web related stuff and it would be good to explore some different areas. If so in what language should I write it?
Sandalot

321jurgen wrote:

So this is maybe a bit far fetched but it would be awesome if you could make a program that runs in the background that downloads and updates all the maps. Maybe I should start programming something myself sometime :/ I only do web related stuff and it would be good to explore some different areas. If so in what language should I write it?
Python would be a good place to start but java would make it a lot more easier for other people to use it.

I had the same idea as you.
Zallius

321jurgen wrote:

So this is maybe a bit far fetched but it would be awesome if you could make a program that runs in the background that downloads and updates all the maps. Maybe I should start programming something myself sometime :/ I only do web related stuff and it would be good to explore some different areas. If so in what language should I write it?
Keeping every map would be quite space consuming, and would also make the game run inefficiently and become a nightmare for reprocessing.
321jurgen

Zallius wrote:

321jurgen wrote:

So this is maybe a bit far fetched but it would be awesome if you could make a program that runs in the background that downloads and updates all the maps. Maybe I should start programming something myself sometime :/ I only do web related stuff and it would be good to explore some different areas. If so in what language should I write it?
Keeping every map would be quite space consuming, and would also make the game run inefficiently and become a nightmare for reprocessing.
That's what I would've thought at first but it will only take like 100gb (I'm guessing) which isn't that much to me and reprocessing (25k beatmaps) only takes like 2 minutes for me and doesn't happen that often.
Topic Starter
RavenMac
Oh goodness, it's been a while. I've just updated the code. One major thing i left out of the code is to set a timeout limit. Personally, i've had the program just sitting at one song or checking one page for hours. Not sure if anyone else has noticed this, because there weren't any messages about this, but it's fixed.

One other thing I've realized, I've used an RSS notification thing for Chrome for some time and have noticed that (either because of the RSS thing or bloodcat) the updates were kinda scattered. It seemed that it wasn't sorted at all and i may have missed an update or new song because of this. So i have gone back on my word and made it check 100 at a time. One problem i can see with this is, it'll want to download 80 songs if you haven't check for 100 songs before. I may add an option if the change between the number of songs last checked and just checked are greater than 20 or somethin like that. Maybe just add the first 100 on the list to start off and keep updating from there? Let me know what yall think.

I've also changed how i save the recently checked items. Instead of using just the IDs, i now just use the entire JSON item array. When comparing with JSON items, if the times for updates are different, then there should have been an update to the song and therefor needs to redownload. And in doing so, I've changed the file used from Recent.num to Recent.json. The program will check to see if you have the .json file first and update, if not it'll check for the .num file and update. In changing the file type, it will delete the Recent.num file and replace it with the .json file.

The last thing i added, and many people probably won't even bother with looking at it, are comments in the code. Basically anything to the right of a # is a comment.



Now for comments. For some reason, i became unsubscribed from my own thread. So i never got notified when yall commented here. Not sure why, but i'm resubscribing again so this shouldn't happen again.

Sir_Sandalot wrote:

RavenMac, you should make it in java. 1+ is a good idea
Also if it sorted all the ranked from the unranked in 2 folders and kept that up to date would be an awesome addition.
I'm workin on it. And (much more likely than with Python) maybe a GUI.

321jurgen wrote:

So this is maybe a bit far fetched but it would be awesome if you could make a program that runs in the background that downloads and updates all the maps. Maybe I should start programming something myself sometime :/ I only do web related stuff and it would be good to explore some different areas. If so in what language should I write it?
Language is all personal preference and what you know. As for doing this, each song added to osu is given an ID. You could see about gettin all of the numbers in a list/array and maybe checking the date created on your computer and comparing it with the last update on osu or bloodcat. I know osu you'll need to get permission to get the API for it, or somethin like that.

Sir_Sandalot wrote:

Python would be a good place to start but java would make it a lot more easier for other people to use it.

I had the same idea as you.
This is the reason why i'd like to make this in Java. That way nobody would have to download anything for it...That is unless they don't have java...

Zallius wrote:

Keeping every map would be quite space consuming, and would also make the game run inefficiently and become a nightmare for reprocessing.
Space consuming? Yes. Game running inefficiently / Reprocessing nightmare? Eh... it depends. Having a ton of songs, i haven't seen much problem playing the game. Preprocessing though, that takes a few minutes though.

321jurgen wrote:

That's what I would've thought at first but it will only take like 100gb (I'm guessing) which isn't that much to me and reprocessing (25k beatmaps) only takes like 2 minutes for me and doesn't happen that often.
I've got a folder of just the .osz files and then the songs folder. My songs folder is over 116GB with 16K song folders and almost 50K maps. Reprocessing takes about 2-3 minutes. It does add up, but i'm a very patient person. And i have 6TB of storage and am looking at upgrading fairly soon. So it's nothing.
Topic Starter
RavenMac
Just updated the code a little bit more. I'm not sure how much detail i should go into what i've changed, so i'll just cover basics. I changed the song info display while downloading. It now shows what number of however many songs you are downloading (downloading 2 of 5). If there's an error, it won't display all of the info again. Just the id artist and title plus the error message.

I changed two functions and added another. One to cut down coding a little. The other was the song info function. Just a slight change. The added function is used to return the song id artist and title to a few parts of the program. It also can return the unicode info if needed.

I've added a little more information in the error messages. It gives a message as to what function the code had an error and then the regular error message.

Not a huge update, but i was mainly hoping to get the display for how many songs are being downloaded finished. Not entirely sure if there will be another update for python, unless asked for.
Sya
Im getting an error: http://puu.sh/bbsFo/8765d8d3ee.png
it immediatly closes the windows after opening the file.
Topic Starter
RavenMac

Syrasu wrote:

Im getting an error: http://puu.sh/bbsFo/8765d8d3ee.png
it immediatly closes the windows after opening the file.
Sorry for the late reply. I was attending a convention.
For some reason when editing the code on Mediafire, it converts the first tab into 4 spaces.
I have to reuploaded the code in order to prevent this.
show more
Please sign in to reply.

New reply