forum

osu!Bancho stats and logs!

posted
Total Posts
129
show more
Topic Starter
Corin

Marcin wrote:

For anyone reading logs from here:
If you want to get rid of the Quits, Joins and Parts message, simply use this program https://dl.dropboxusercontent.com/u/211 ... leaner.exe
Drag and drop a text file over this program, and it'll create clean text file. Useful for reading logs from #osu.
I already have the regex and the cleaner built.
I'm building a php wrapper to show the logs and have a date selector too along with fixes for UTF-8.

Here http://osu.corin.me/logs/temptest
Anything in that dir has been scrubbed.

Once the wrapper is finished, I'll scrub them all and build a script that rsyncs the files and scubs it automatically.
Marcin
Btw; last log from #polish for ex. is from #polish_20130604.log ... is everything working fine?
Topic Starter
Corin
Zertap pointed it out to me that the rsync had stopped working.

Just realized I forgot to uncomment the rsync cronjob >.>

All should be good now
silmarilen
can you do an update on all the channels? some are gone and there are new ones aswell
Topic Starter
Corin

silmarilen wrote:

can you do an update on all the channels? some are gone and there are new ones aswell
Will do.


The log viewer will be out by the end of today/in a few hours.

Just talking to the guy about it now.
Topic Starter
Corin
All done! It looks ugly for now but
http://osu.corin.me/logs/
It works :D

Just swapping out the rsync script to add it to specific folderz
Rei Hakurei
umm can you give "automated" avatar support? (well, you can't automate it for "underscore" and "space" based name... D:)
http://s.ppy.sh/a/UserID
(or something)
which UserID can be obtained from the user profile page itself
  while(!(EOF)) {
%line = %fo->$readLine
if ($str.match("^var userId = *",%line,r)) {
%uid = $int($str.chop($str.mid(%line,13,$length(%line)),2))
// this is not php or what code, picked from KVIrc
break;
}
}


(if cant, can you "re-register" me?)
Topic Starter
Corin

Rei Hakurei wrote:

umm can you give "automated" avatar support? (well, you can't automate it for "underscore" and "space" based name... D:)
http://s.ppy.sh/a/UserID
(or something)
which UserID can be obtained from the user profile page itself
  while(!(EOF)) {
%line = %fo->$readLine
if ($str.match("^var userId = *",%line,r)) {
%uid = $int($str.chop($str.mid(%line,13,$length(%line)),2))
// this is not php or what code, picked from KVIrc
break;
}
}


(if cant, can you "re-register" me?)
....... and this is actually what I needed.

I'll get this shoved in today.
Topic Starter
Corin
Aaand completely forgot about the scripts, should be fine now.
Nanika133
Can you make stats for other channels? That would be really nice of you.
Topic Starter
Corin

Nanika133 wrote:

Can you make stats for other channels? That would be really nice of you.
Am I missing any? It should be generating stats for all channels... Have a look at http://osu.corin.me/stats

Let me know if I'm not logging a channel or two.
UnderminE
Turkish ;_;
Nanika133
#arabic is not there, #bulgarian #cantonese #chinese are missing too...
Topic Starter
Corin

UnderminE wrote:

Turkish ;_;

Nanika133 wrote:

#arabic is not there, #bulgarian #cantonese #chinese are missing too...
Done and done, they are currently generating so it will take a little while to go through all 250 odd days.

Also added #english too.

EDIT: All done.
Mirailine
#romanian is also missing.
UnderminE

Corin wrote:

UnderminE wrote:

Turkish ;_;

Nanika133 wrote:

#arabic is not there, #bulgarian #cantonese #chinese are missing too...
Done and done, they are currently generating so it will take a little while to go through all 250 odd days.

Also added #english too.

EDIT: All done.
Thankies <3
Topic Starter
Corin

Mirailine wrote:

#romanian is also missing.
Done: http://osu.corin.me/stats/romanian.html
Nanika133
Yeah, thanks alot :3
silmarilen
i dont know which channel this shows http://osu.corin.me/stats/english.html but its not #english
Topic Starter
Corin

silmarilen wrote:

i dont know which channel this shows http://osu.corin.me/stats/english.html but its not #english
Logfile = "/logops/osu/#ctb_*.log"

whoops, should be fixed.
UnderminE
You know what, we should link these stat pages in channels as an auto/start message. Like when you open a channel, there will be an auto-message with an explanation and the stats page-link. It would bring some activity within IMO. Activity is not a problem for crowded channels (#osu, #modhelp etc) but less-crowded ones.
Topic Starter
Corin
I could make a bot for that but it's down to peppy if he'd let me make it.

If he built it into Bancho as it does with the messages when a user joins a channel it'd be far more easier however I can make a bot that does all kinds of shit IE can PM you the link to the stats/logs of the channel you're in just by doing .stats or .log in a channel.

But I think peppy said no to the last one.
UnderminE
Well last one is the same as coming and finding the link here, I suggested it for new players (mostly foreign) who don't browse forums at all.

Maybe we can ask peppy about the first one.
Secretpipe
I think the website is down :/
Topic Starter
Corin
Nope? Working fine here.
YERTI

Corin wrote:

Nope? Working fine here.
i think he meant that the stats arent updating
Topic Starter
Corin
all fixed, I moved a lot of shit around and reorganized stuff so there was a permdenied issue.
Patched and working once more~
bigbadwolf666_old
user avater don't work (at least for #italian stats)
silmarilen
the logs dont seem to work anymore, im just getting this when i try to select a channel
Topic Starter
Corin
php broke, I was pissing about with php execution in certain dirs and forgot about that one >.>

Should work fine now.
- Marco -


this don't work :c
Saten
any idea why the logs in skandinavian doesn't work?
Juuryoushin
Seems like all logs/quotes are completely freezed and not updated since Wednesday. (there are no logs from Thursday and today)
Did something bad happened? :I

EDIT: okay nvm
Topic Starter
Corin
It's temporarily not generating stats, the logs are fine juts not being sync'd to the log folders, I need to work on a few things on it because at the moment it's pushing full load on my servers CPU and I have an inkling that the HDD is on its last legs due to constant IO.

Anyways once I finish my build-scripts I'll finish the stats stuff, but I'll re-enable log syncing for now.

Saten wrote:

any idea why the logs in skandinavian doesn't work?
It doesn't? o.O

EDIT: Oh re-read
Topic Starter
Corin
Herro!

So what I did to counter this issue is put more load on it to ease off the big burden!

So there's a few new things; one is that that stats are now split into 2 sections.
Yearly and combined.

Combined will be filed in the same place as normal however they will be generated once a day only. Not every hour, this will generally ease off the load

Yearly will be filed in http://osu.corin.me/stats/yearly/ so you can view the past stats of 2013 and this years current stats.
Before you moan that /np counts as an action, I'm aware I just can't tell what changed.

2014's yearly stats will now be generated every 10 mins once again but is more likely to be changed to 20 in the near future.

Logs have started syncing once again as normal aaand I think that's all! Enjoy :D
Juuryoushin
And all the avatars are back !
Grats \:D/
Topic Starter
Corin

Didou wrote:

And all the avatars are back !
Grats \:D/
Yeup, automatically adding themselves and updating, there's a few glitches here and there but the majority of it works!
Topic Starter
Corin
Did a quick but fairly decent modification
Thanks to peppy pointing out to me that the osu!API can get uid's; avatars no longer are cached locally on the server but use your current osu! one directly so it'll never be out of date!

This will lower load and generation times greatly meaning I can re-enable #osu's combined stats generation!

#osu's 2014 statistics will be generated hourly
#osu's combined statistics will now be generated daily rather than whenever it feels like it.

Have fun!

Also if anyone wants it a copy of the UID cache can be found here: http://corin.me/uid_cache It's always up to date.
Format is Username:UID
seasonS

Vext wrote:

Pretty nifty. If only I ever spoke in #osu or anywhere.
True
Topic Starter
Corin
So I did a few changes which is why this was down for a few days.

1: I moved everything to a new dedi so now instead of #osu taking 3 hours and 10 mins to generate on combine stats and dragging a load of other shit down because of; it takes 10 mins and drags nothing down with it. 2x E5's are nice.
So now I once more re-enabled #osu combined stats.

2: I redid the way I have logs laid out. While moving everything I somewhat was confused and had to ask myself what the fuck I was doing.
Every night it would rsync the logfiles over to another directory. So it used to copy them and overwrite, for some reason rsync didn't like checking if the files were the exact same in both destinations so what it did was overwrote them ALL so that's every channel, copying 500 files ($chancount * 500) every night which I'm sure the HDD did not like.
It's a very dumb thing to do, especially seeing as the old dedi itself was not very powerful and low on HDD space (45GB total in logs * 2 due to them being in 2 locations).

So now I replaced that shit with something much more simplistic. I created a symlink to the actual files and every night at 12.01AM it'll create a symlink for that day bringing the overall HDD load to 0 and bringing down space requirements.

This also gave another boon; the logs are now updated in realtime so you don't have to wait an hour for that stuff to update anymore.
Looking back I knew how to do all this back then and I'm wondering why I didn't do it. Oh well.

Anyways, enjoy. \o\ Everything should be overall faster

TL;DR
Chan stats generate zoom zoom now
Chan stats are back up
#osu combined now works
Logs are updated in realtime
show more
Please sign in to reply.

New reply