Every time I visit this site, I can't help but notice the '4.xx billion ranked plays!' notice at the upper right. I know that each of those plays has millisecond resolution of exactly what the player did, the stimulus they received, etc - and of course there's also a record of exactly how people's play changes over time, how they improve, etc. It strikes me as an incredible resource for understanding how humans deal with timing, how muscle memory develops, not to mention changes in reading skill and things like that. It also seems like it'd be an incredible set of data for doing some kind of machine learning on - see if you can predict what points a player is likely to slip up on, or even how differences in presentation could actually make it easier or harder to correctly play the sequence - what's the relationship between shape and timing, etc.

Obviously there are some problems on the side of the researchers as well (for example, the way that osu! collects and also assigns data would probably not fit with the institutional rules for human experimentation and I don't actually know what sorts of rules there'd be about making use of data collected by an outside company). But it seems like there's so much potential here, it must be worth working out an appropriate way to do it.

I guess a middle ground would be to actually develop predictors and things like that for use by the osu! software itself - something that can give players feedback about how they're learning, what maps might help them improve fastest, or even have some kind of 'teaching mode' which manipulates the display subtly on the fly to try to help the player improve their timing (for example, poking around in my own data, I've found that timing errors are actually somewhat predictable based on the last few note timings, and I could see that even just using about 1000 notes)

Is this something people would be interested in? Or would it feel like an intrusion or a violation of privacy? What do you think?

I am already researching that as part of the osu!skill project. I already developed models of note-to-note reading and such, and am in the process of modeling the players' reading base on systems of notes. The end goal is to be able to analyze replay and generate probabilities based on which skill aspect the player screwed up on. You don't really need machine learning to figure this out, and if you do, then you don't really understand how everything works.

Obviously there are some problems on the side of the researchers as well (for example, the way that osu! collects and also assigns data would probably not fit with the institutional rules for human experimentation and I don't actually know what sorts of rules there'd be about making use of data collected by an outside company). But it seems like there's so much potential here, it must be worth working out an appropriate way to do it.

Is this something people would be interested in? Or would it feel like an intrusion or a violation of privacy? What do you think?

Hilarious. There is no intrusion of privacy if the stuff is available for access. The osu!api lets you gather player replays, scores and all of such. If you are violating anything, peppy will message you and tell to take it down. That's as serious as it can possibly get.

It did occur to me that one could just go and do it, but it's always better to ask and feel out the community response I think. People's feelings tend to be complex about this kind of thing, and especially with data analysis there's a sort of 'I didn't know it could be used for that!' response that often happens.

Anyhow, for example, I know people in the area who are interested in human time perception, and they use the statistics (measured from a relatively small set of subjects) to distinguish between cognitive models.So osu! replays could be a gigantic resource for that kind of research - drop the error bars by 4 orders of magnitude instantly. I also know people who are studying learning and they have a certain metric they invented to detect when someone has learned something separately or as a group - so you could basically look to see at what point people go from reading 'don' 'don' 'kat' as separate beats to reading ddk as a pattern. Those are just some examples based on people I've personally met, but I imagine there'd be all sorts of other ideas that could spring up around this kind of data.

For the machine learning side, the thing I was messing with was to build a me-controller. That is to say, have the computer basically play a stimulus to me in order to manipulate my timing on the next beats, based on what I did on the previous beats. In my own experiments outside of osu!, I was able to induce a small improvement in accuracy while streaming dk, but it was very unstable and basically fell apart after 100 notes. That was just using an SVM autoregressor and a 4 beat history, so quite primitive compared to other things one might do (but which require more data to avoid overfitting). Incidentally, timing error in dk streaming tended to alternate for me - when I was early for the kat, I was late for the don, and vice versa. Also, that feeling of losing control when you mess up a whole bunch of notes in a row seems like it has a growing signal 20-30 beats before you actually start to screw up - at least for me, the signature was a drift in mean timing offset.

The osu!skills thing would be another place I could see it being useful - predicting a player's level from a play sequence or something - but I guess you're doing it another way (sounds like you're doing something like generating a set of derived features and then doing regression on top of the derived features to predict performance on unseen maps?)

I think if you were able to predict which plays correspond to the highest chance of a change in skills, that'd also be nice - basically a map recommender to help you learn as fast as possible.

You could also go exploratory and go look for patterns in play. Another thing that might be interesting at the intersection between neuroscience and machine learning would be to see if you can predict demographic information from the play sequences (for example, age, nationality, etc - whatever people have put on their profiles). Of course a lot of the demographic data on the profiles is fake stuff, so it'd be a ton of work to clean that up. Either way, I think there's a lot of stuff one might do given this data.

Anyhow, I did look at osu!api, but I think it really wasn't designed for this kind of thing. For this, you'd want to basically just get a large chunk of the replays and then once you had them, you wouldn't need to transfer more data. So the example of use would be to basically set it up to pull the full 10/minute replays and just let it alone for a month. But I don't think that was the intended use pattern. So if it looks like there's something that could come of this beyond just speculation, I'd want to just try to get in contact with peppy directly and figure out the best way to transfer that kind of data.

NichG wrote:

It did occur to me that one could just go and do it, but it's always better to ask and feel out the community response I think. People's feelings tend to be complex about this kind of thing, and especially with data analysis there's a sort of 'I didn't know it could be used for that!' response that often happens.

If a day comes when someone says 'I didn't know it could be used for that!' for such a thing, then someone who is NOT you screwed up somewhere. Any data made available is free for public use, and nothing in the osu! terms of service say that this data has restricted use (to my knownledge). Heck, peppy's anti-cheat system has privacy concerns much more questionable than whatever you might do with the replay data.

Anyhow, for example, I know people in the area who are interested in human time perception, and they use the statistics (measured from a relatively small set of subjects) to distinguish between cognitive models.So osu! replays could be a gigantic resource for that kind of research - drop the error bars by 4 orders of magnitude instantly. I also know people who are studying learning and they have a certain metric they invented to detect when someone has learned something separately or as a group - so you could basically look to see at what point people go from reading 'don' 'don' 'kat' as separate beats to reading ddk as a pattern. Those are just some examples based on people I've personally met, but I imagine there'd be all sorts of other ideas that could spring up around this kind of data.

That would be useful to figuring out how the reading skill should work. What metric does it use?

NichG wrote:

For the machine learning side, the thing I was messing with was to build a me-controller. That is to say, have the computer basically play a stimulus to me in order to manipulate my timing on the next beats, based on what I did on the previous beats. In my own experiments outside of osu!, I was able to induce a small improvement in accuracy while streaming dk, but it was very unstable and basically fell apart after 100 notes. That was just using an SVM autoregressor and a 4 beat history, so quite primitive compared to other things one might do (but which require more data to avoid overfitting). Incidentally, timing error in dk streaming tended to alternate for me - when I was early for the kat, I was late for the don, and vice versa. Also, that feeling of losing control when you mess up a whole bunch of notes in a row seems like it has a growing signal 20-30 beats before you actually start to screw up - at least for me, the signature was a drift in mean timing offset.

You lost me after the "have the computer basically play a stimulus to me" part.

NichG wrote:

The osu!skills thing would be another place I could see it being useful - predicting a player's level from a play sequence or something - but I guess you're doing it another way (sounds like you're doing something like generating a set of derived features and then doing regression on top of the derived features to predict performance on unseen maps?)

More or less, that's the goal

NichG wrote:

You could also go exploratory and go look for patterns in play. Another thing that might be interesting at the intersection between neuroscience and machine learning would be to see if you can predict demographic information from the play sequences (for example, age, nationality, etc - whatever people have put on their profiles). Of course a lot of the demographic data on the profiles is fake stuff, so it'd be a ton of work to clean that up. Either way, I think there's a lot of stuff one might do given this data.

One of my ideas on how to deal with cheated plays deals with the neuroscience you are talking about. Having a system that analyzes replays and generates a player's neural "fingerprint" is a topic a plan to research when and if I finish osu!skills.

NichG wrote:

Anyhow, I did look at osu!api, but I think it really wasn't designed for this kind of thing. For this, you'd want to basically just get a large chunk of the replays and then once you had them, you wouldn't need to transfer more data. So the example of use would be to basically set it up to pull the full 10/minute replays and just let it alone for a month. But I don't think that was the intended use pattern. So if it looks like there's something that could come of this beyond just speculation, I'd want to just try to get in contact with peppy directly and figure out the best way to transfer that kind of data.

Good luck with that. If it's in peppy's interest (like cheat detection), he may give you a set of confirmed cheated plays. Not sure about a total database.

For the separate/group thing, I'm not sure the exact details, but it was using something about the variance pattern. I think this might be the paper: https://scholar.google.com/citations?vi ... sOgNQ5qMEC
Roughly what I'd do to adapt the method is to look at a particular test pattern like ddk and ask whether the variance in timing error is higher at the edges of the pattern, or if its relatively uniform throughout.

For the me-controller thing, I allowed the computer to change the timing of the kats while keeping the dons at a regular pace. So the computer would basically 'force' me to make an error if I were playing perfectly, but it forced me to make the error which would make my timing error on the dons equal to zero according to the linear model. It worked a little bit, but not very well - it became unstable after awhile and was doing really weird things.

For the cheated plays, unfortunately I think those are the least scientifically interesting part of this database.